Surgical applications with integrated visualization camera and optical coherence tomography

ABSTRACT

A system for guiding an ophthalmic procedure is disclosed. The system includes a housing assembly with a head unit configured to be at least partially directed towards a target site in an eye. An optical coherence tomography (OCT) module and stereoscopic visualization camera are at least partially located in the head unit and configured to obtain a first set and a second set of volumetric data, respectively. A controller is configured to register the first set and second set of volumetric data to create a third set of registered volumetric data. The third set and second set of registered volumetric data are rendered, via a volumetric render module, to a first and second region. The first region and the second region are overlaid to obtain a shared composite view of the target site. The controller is configured to extract structural features and/or enable visualization of the target site.

PRIORITY CLAIM

This application claims priority to, and benefit of, U.S. ProvisionalApplication No. 62/943,965, filed on Dec. 5, 2019, the content of whichis hereby incorporated by reference in its entirety.

INTRODUCTION

The present disclosure relates to a system for guiding an ophthalmicprocedure, the system having an integrated visualization camera andoptical coherence tomography module. Various imaging modalities may beemployed to assist a surgical team prior to and during ophthalmicsurgery. Each of these imaging modalities brings a different set ofinformation to the table, each presenting with a unique set of issues.For example, in a corneal transplant, attempting to view the transplantthrough a microscope is difficult due to thetransparent/semi-transparent property of the corneal tissue. When a dyeis used, the dye may not stain the tissue in a way that assistsvisualization; for example, the dye might emphasize edges or otherdefects but may not sufficiently emphasize a folded, torn or wrinkledarea. In order to maximize the available information, a synthesis ormelding of the respective information provided by various imagingmodalities is desirable. However, it is no trivial matter to preciselyrepresent, to a user, the captures made by multiple imaging modalitiesas if they represent the exact same three-dimensional object in alldimensions. It is further challenging to extract useful information fromthe multiple imaging modalities during an ophthalmic procedure inreal-time.

SUMMARY

Disclosed herein is a system for guiding an ophthalmic procedure. Thesystem includes a housing assembly having a head unit configured to beat least partially directed towards the target site. An opticalcoherence tomography (OCT) module is at least partially located in thehead unit and configured to obtain a first set of volumetric data of thetarget site. A stereoscopic visualization camera is at least partiallylocated in the head unit and configured to obtain a second set ofvolumetric data of the target site. The second set of volumetric dataincludes first and second (e.g. left and right) views of the targetsite.

The system enables improved imaging, visualization and extraction ofboth structural features and pathologies and for retinal, corneal,cataract and other ophthalmic surgeries. The system is movable and maybe implemented as a diagnostic imaging system and/or an ophthalmicsurgical system. The system results in improved patient outcomes in manyfields, such as ophthalmology, by fusing various datasets related to thepatient and the surgical site. The datasets may include but are notlimited to: stereoscopic visualization of the surgical site; opticalcoherence tomography of the patient's eye in whole, or in part. Thedatasets may include other volumetric scanning techniques, such asultrasound and magnetic resonance imaging and one or more refractivemodels of the eye, which may be generated using eye characterizationtechniques.

The system includes a controller in communication with the stereoscopicvisualization camera and the OCT module. A volumetric render module isselectively executable by the controller and/or a camera processorintegrated within the stereoscopic visualization camera. The controllerhas a processor and tangible, non-transitory memory on whichinstructions are recorded. The controller is configured to register thefirst set of volumetric data from the OCT module with the second set ofvolumetric data from the stereoscopic visualization camera to create athird set of registered volumetric data. The third set of registeredvolumetric data is rendered, via the volumetric render module, to afirst region to obtain a two-dimensional OCT view. The second set ofvolumetric data from the stereoscopic visualization camera is rendered,via the volumetric render module, to a second region to obtain a livetwo-dimensional stereoscopic view.

The first region and the second region are overlaid to obtain a sharedcomposite view of the target site, with the controller being configuredto visualize and/or extract features of the target site from the sharedcomposite view. In another embodiment, the system includes avisualization camera at least partially located in the head unit andconfigured to obtain a second set of two-dimensional image data ortwo-dimensional image of the target site, the two-dimensional imageincluding first and second views of the target site. The controller isconfigured to register the first set of volumetric data from the OCTmodule with the image data from the visualization camera to create athird set of registered volumetric data. The third set of registeredvolumetric data is rendered to a first region to obtain amulti-dimensional OCT view, via a volumetric render module; selectivelyexecutable by the controller. The second set of two-dimensional imagedata from the visualization camera is rendered to a second region toobtain a live multi-dimensional view, via the volumetric render module.The first region and the second region are overlaid to obtain a sharedcomposite view of the target site.

In one example, the controller is configured to obtain a plurality ofdepth scans extending through a corneal surface, with each of theplurality of depth scans defining respective starting points. A pointcloud is generated or collected from the respective three-dimensionallocations corresponding to the respective starting points of theplurality of depth scans. The point cloud is converted to obtain anextracted curvature. Obtaining the extracted curvature may includeinterpolating between the respective starting points.

The extracted curvature may be characterized by a plurality of depths.In one example, the controller is configured to visualize the sharedcomposite view with a plurality of topographic levels. The plurality oftopographic levels respectively represent the plurality of depths suchthat the extracted curvature may be visualized.

In another example, the ophthalmic procedure is a cataract surgeryincluding implantation of an intraocular lens into the eye. Thecontroller may be configured to add at least one annotation over theshared composite view on a display such that the annotation indicates aportion of the extracted curvature. The relative position of theannotation in the shared composite view may be maintained, and theextracted curvature may be used to guide alignment of the intraoculardevice to the eye. The first set of volumetric data is configured to beupdated at a first frequency and the second set of volumetric data isconfigured to be updated at a second frequency. The updating of thefirst set of volumetric data and the second set of volumetric data maybe synchronized to facilitate the alignment of the intraocular device tothe eye.

In yet another example, the controller may be configured to obtainrespective axial length measurements in real time repeatedly during theophthalmic procedure by switching the OCT module between a firstresolution mode and the second resolution mode.

In yet another example, the ophthalmic procedure is a cornealtransplant. The controller may be configured to obtain a plurality ofdepth scans of the cornea. The controller is configured to identify andisolate a pathological region as being between a first one of theplurality of depth scans and a second one of the plurality of depthscans. The controller is configured to add at least one annotation overthe shared composite view on a display, the annotation indicating thepathological region.

In yet another example, the ophthalmic procedure includes astigmatismcorrection. The controller may be configured to obtain a plurality ofrow scans of the cornea. The controller may be configured to extract asteep meridian and a flat meridian from the plurality of row scans, viatracking of respective maximum and respective minimum points ofcurvature on the cornea. The plurality of row scans may be arranged in astar pattern.

The system may include a robotic arm operatively connected to andconfigured to selectively move the head unit. The robotic arm isselectively operable to extend a viewing range of the OCT module in anaxial direction, a first transverse direction and a second transversedirection. Registering the first set of volumetric data from the OCTmodule with the second set of volumetric data from the stereoscopicvisualization camera may include: aligning the first and second views ofthe stereoscopic visualization camera respectively in rotation,translation and scale to the volumetric render module; and matching therespective perspectives of the first and second views of thestereoscopic visualization camera to the volumetric render module.

Prior to registering the first set of volumetric data with the secondset of volumetric data, the controller is configured to obtain atransformation matrix connecting the respective space of the OCT moduleto the respective space of the volumetric render module. Prior toregistering the first set of volumetric data with the second set ofvolumetric data, the controller may be configured to calibrate the OCTmodule and calibrate the stereoscopic visualization camera. Registeringthe first set of volumetric data with the second set of volumetric datamay include finding a respective location and respective orientation ofa center of projection of first and second two-dimensional visualizationmodules of the stereoscopic visualization camera relative to therespective location and the respective orientation of a respective dataspace of the OCT module.

Registering the first set of volumetric data with the second set ofvolumetric data may include aligning a local area of interest in thefirst set of volumetric data in position, orientation and size with thesecond set of volumetric data. The local area of interest may include atleast one of a corneal limbus and a scleral vasculature.

The above features and advantages and other features and advantages ofthe present disclosure are readily apparent from the following detaileddescription of the best modes for carrying out the disclosure when takenin connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic fragmentary perspective view of a system having anintegrated stereoscopic visualization camera and optical coherencetomography (OCT) module;

FIG. 2 is a schematic diagram of a portion of the system of FIG. 1, thesystem having a controller;

FIG. 3 is a flowchart of an example method implemented by the system ofFIGS. 1 and 2 to obtain a shared composite view based on thestereoscopic visualization camera and optical coherence tomography (OCT)module;

FIGS. 4A and 4B are schematic fragmentary perspective views of examplescanning regions for the OCT module of FIG. 1;

FIGS. 4C and 4D are schematic fragmentary top views of example scanningpatterns for the OCT module of FIG. 1;

FIG. 5 is a schematic fragmentary view of an example shared compositeview showing a live image of an eye overlaid with a topographic outline;

FIG. 6A is schematic illustration of an axial calibration set-up for theOCT module of FIG. 1;

FIGS. 6B, 6C and 6D respectively illustrate a top view, a side view anda front view of an OCT axial calibration device;

FIGS. 7A and 7B illustrate example data received from the OCT axialcalibration device shown FIGS. 6B, 6C and 6D;

FIG. 8A is a schematic top view of a first example of a calibrationtarget;

FIG. 8B is a schematic top view of a second example of a calibrationtarget;

FIG. 9A is a schematic perspective view of a set-up with a plurality ofcalibration targets for transverse calibration of the OCT module;

FIGS. 9B and 9C illustrate example data received from the set-up of FIG.9A;

FIG. 10A is a schematic perspective view of another set-up fordetermining the location and orientation of respective data space of theOCT module relative to a calibration device;

FIG. 10B is a schematic front transverse view of example data receivedfrom the set-up of FIG. 10A;

FIGS. 11A and 11B are schematic perspective views illustrating locationand orientation of the calibration device (of FIG. 10A) relative to oneof two monoscopic views of the stereoscopic visualization camera;

FIG. 12A is a schematic perspective view illustrating location andorientation of the respective data space of the OCT module relative toone of two monoscopic views of the stereoscopic visualization camera;

FIG. 12B is a schematic diagram illustrating the inverse relationship ofa plurality of points;

FIG. 13A is a schematic fragmentary top down view of a shared compositeview of an eye;

FIGS. 13B and 13C are schematic fragmentary oblique views of the eye ofFIG. 13A, in accordance with a first oblique angle and a second obliqueangle, respectively;

FIG. 14A is a schematic fragmentary perspective view of the eye showinga plurality of depth scans of the eye;

FIG. 14B shows a schematic fragmentary sectional view of the eye, alongwith schematic diagrams of output images obtained from the plurality ofdepth scans of FIG. 14A;

FIG. 15 is a schematic fragmentary sectional view showing the use of thesystem to extract a three dimensional volumetric representation of theeye;

FIG. 16A is a schematic diagram of an output image of a single slice ofthe eye;

FIG. 16B is a schematic fragmentary perspective view of the eye,illustrating multiple slices across the surface of the eye;

FIG. 16C is a schematic diagram of a superimposed image showing theoutputs obtained from the multiple slices of FIG. 16B;

FIG. 16D is a schematic diagram showing the curvature of the corneaextracted through respective points on the multiple slices of FIG. 16B;

FIG. 17A is a schematic fragmentary perspective view of the eye, showinga plurality of depth scans arranged in a star pattern;

FIG. 17B is a schematic fragmentary perspective view of the eye,illustrating a steep axis and a flat axis of an astigmatic cornea,extracted through the plurality of depth scans of FIG. 17A;

FIG. 18A is a schematic diagram of an intraocular lens configured forimplantation into an eye;

FIG. 18B is a schematic diagram of a shared composite view of an eyesuperimposed with the intraocular lens of FIG. 18A, the eye and theintraocular lens not being sufficiently aligned;

FIG. 18C is a schematic diagram of the shared composite view of FIG.18B, with the curvature of the live surfaces extracted via the systembeing used to guide the placement of the intraocular lens and improvealignment; and

FIG. 19 is a schematic diagram of a shared composite view during acorneal procedure, showing visualization of a pathological region in theeye in real-time.

DETAILED DESCRIPTION

Referring to the drawings, wherein like reference numbers refer to likecomponents, FIGS. 1 and 2 schematically illustrate a system 10 having avisualization camera 12, which may be a stereoscopic visualizationcamera 12. The visualization camera 12 may include other types ofmultidimensional imaging devices. The system 10 includes an opticalcoherence tomography module 14, referred to hereinafter as “OCT module”14. The system 10 is configured to guide an ophthalmic procedure on atarget site 16 in a patient's eye. Referring to FIG. 1, the stereoscopicvisualization camera 12 and OCT module 14 are at least partially locatedin a head unit 18 of a housing assembly 20, with the head unit 18configured to be at least partially directed towards the target site 16.As described below, the OCT module 14 is configured to obtain a firstset of volumetric data of the target site 16 while the stereoscopicvisualization camera 12 is configured to obtain a second set ofvolumetric data of the target site 16. The stereoscopic visualizationcamera 12 is configured to record first and second images of the targetsite 16, via first and second 2D visualization modules V1, V2, togenerate a live two-dimensional stereoscopic view of the target site 16.By overlaying volume-rendered OCT data onto the live two-dimensionalstereoscopic view of the stereoscopic visualization camera 12, thesystem 10 enables the detection of and response to various pathologiespresent in tissue.

I. System Components

Referring to FIG. 1, at least one selector 22 may be mounted on the headunit 18 for selecting specific features, such as magnification, focusand other features. The selector 22 may be employed to enable anoperator to manually position the head unit 18. The system 10 mayinclude a robotic arm 24 operatively connected to and configured toselectively move the head unit 18. For example, referring to FIG. 2, therobotic arm 24 may be selectively operable to extend a viewing range ofthe OCT module 14 in an axial direction A, a first transverse directionT1 and a second transverse direction T2.

Referring to FIG. 1, the head unit 18 may be mechanically coupled to therobotic arm 24 via a coupling plate 26. The coupling plate 26 mayinclude one or more joints configured to provide further degrees ofpositioning and/or orientation of the head unit 18. The head unit 18 maybe connected to a cart 30 having at least one display medium (which maybe monitor, terminal or other form of two-dimensional visualization),such as first and second displays 32 and 34 shown in FIG. 1. The housingassembly 20 may be self-contained and movable between various locations.Returning to FIG. 1, the first display 32 may be connected to the cart30 via a flexible mechanical arm 36 with one or more joints to enableflexible positioning. The flexible mechanical arm 36 may be configuredto be sufficiently long to extend over a patient during surgery toprovide relatively close viewing for a surgeon. The first and seconddisplays 32 and 34 may include any type of display including ahigh-definition television, an ultra-high definition television,smart-eyewear, projectors, one or more computer screens, laptopcomputers, tablet computers, and/or smartphones and may include atouchscreen.

Referring to FIGS. 1 and 2, the system 10 includes a controller C havingat least one processor P and at least one memory M (or non-transitory,tangible computer readable storage medium) on which are recordedinstructions for executing method 100, shown in and described below withrespect to FIG. 3. The memory M can store controller-executableinstruction sets, and the processor P can execute thecontroller-executable instruction sets stored in the memory M. Referringto FIG. 1, the controller C may be housed in the cart 30 and configuredto control the robotic arm 24. The controller C may be configured toprocess signals for broadcasting on the first and second displays 32 and34.

Referring now to FIG. 2, a schematic view of a portion of system 10 isshown. The head unit 18 is configured to house at least some portion ofthe OCT module 14 and at least some portion of the stereoscopicvisualization camera 12. In one example, a first portion 14A of the OCTmodule 14 is housed in the head unit 18 while second portion 14B is not.Similarly, a first portion 12A of the stereoscopic visualization camera12 is housed in the head unit 18 whereas second portion 12B is not.Referring to FIG. 2, the OCT module 14 includes a first light source 40,a beam splitter 42, a detector 44, a reference arm 46 and a sample arm48. In one example, the detector 44 includes a spectrometer. However, itis understood that the detector 44 may include other types of receptordevices available to those skilled in the art.

The OCT module 14 and the stereoscopic visualization camera 12 mayinclude integrated processors in communication with the controller C.For example, referring to FIG. 2, the OCT module 14 may include an OCTprocessor 50 and the stereoscopic visualization camera 12 may include acamera processor 52. The OCT processor 50 and camera processor 52 may beseparate modules in communication with the controller C. Alternatively,the OCT processor 50 and the camera processor 52 may be embedded in thecontroller C. The camera processor 52 and/or the controller C areconfigured to selectively execute a volumetric render module 51,referred to hereinafter as “VR module,” and two-dimensional stereoscopicvisualization modules V1, V2. The VR module 51 may be employed withstereoscopic and non-stereoscopic data.

By integrating the OCT module 14 with the stereoscopic visualizationcamera 12, the system 10 enables much more immersive viewing andinteraction with the image captures from the two modalities. The VRmodule 51 is used to display the three-dimensional data stereoscopicallyon a stereoscopic display. In one example, the VR module 51 may bemodeled as two monoscopic volume renders separated in a horizontaldirection by some intraocular distance and converging at a desiredpoint, with some additional constraints such as the two views having thesame boundaries at the focal plane of the system. For accurate fusion,the intraocular distance and convergence distance of the 2Dvisualization modules V1, V2 is input to the VR module 51 to achieveidentical stereoscopic parameters between the two modalities.

Referring to FIG. 2, the controller C and/or the OCT processor 50 and/orthe camera processor 52 may be in communication with a user interface 54and the first and displays 32, 34, via a short-range network 56. Theshort-range network 56 may be a bi-directional bus implemented invarious ways, such as for example, a serial communication bus in theform of a local area network. The local area network may include, but isnot limited to, a Controller Area Network (CAN), a Controller AreaNetwork with Flexible Data Rate (CAN-FD), Ethernet, blue tooth, WIFI andother forms of data connection. The short-range network 56 may be aBluetooth™ connection, defined as being a short-range radio technology(or wireless technology) aimed at simplifying communications amongInternet devices and between devices and the Internet. Bluetooth™ is anopen wireless technology standard for transmitting fixed and mobileelectronic device data over short distances and creates personalnetworks operating within the 2.4 GHz band. Other types of connectionsmay be employed.

Referring to FIG. 2, the beam splitter 42 is configured to splitincoming light L1 from the first light source 40 and send itsimultaneously along a reference light path L2 into the reference arm46, as well as along a sample light path L3 into the sample arm 48. Thesample arm 48 is configured to direct at least a portion of the beamoriginating from the first light source 40 (split by the beam splitter42) onto the target site 16. Referring to FIG. 2, the target site 16 isilluminated by a first beam B1. Fiber optics may be employed totransport and/or guide the first beam B1 and direct it to fall in theform of a spot scan 60 onto an appropriate region of interest in thetarget site 16. Other methods available to those skilled may be employedto transport and/or guide the beams within the various components of thesystem 10.

Referring to FIG. 2, the target site 16 onto which the spot scan 60 isdirected to fall may include structures which at least partially reflectincident light arriving along first beam B1, resulting in a firstreflected beam R1. The reference arm 46 is configured to reflect alongthe reference light path L2 at least a portion of the first light source40 that had been split by the beam splitter 42 and sent along thereference light path L2. The reference arm 46 may include a referencearm reflecting device 62, such as a mirror or corner cube, placed at aselected distance along the reference light path L2 and configured toselectively reflect light of a specific desired wavelength. referencearm reflecting device 62 may be oriented relative to the reference lightpath L2 such that a relatively large portion, for example 95% or more,of the wavelengths incoming along the reference light path L2 arereflected along the reference light path L2, back towards the beamsplitter 42.

Referring to FIG. 2, the beam splitter 42 is configured to opticallycombine the light reflected from the reference arm 46 (along referencelight path L2) and the sample arm 48 and send the resulting combinedbeam L4 to detector 44. The respective optical lengths or traveldistance of the light in the reference arm 46 and sample arm 48 areconfigured to match (reference light path L2 and sample path L3) suchthat the interference of the light reflected back from the reference arm46 and the sample arm 48 encodes the location of multiple reflectionpoints in the target site 16 relative to some known reference point orrelative to each other. The encoding may be captured by a line-scancamera in the detector 44 and processed through spectral and Fourieranalysis, via the OCT processor 50 and/or the controller C.

Referring to FIG. 2, the incoming light from the sample light path L3 isdirected via a steering unit 64, downward through an optical deflectingelement 70, and subsequently through a common objective lens set 72 andinto the target site 16, where it strikes the target site 16 and isreflected back along the first reflected light R1. The steering unit 64may include plurality of steering members, such as first steering member65, second steering member 66 and third steering member 68. The incominglight from the sample light path L3 may be coupled optically with thehead unit 18 via optical connector 74 and interface 76. The commonobjective lens set 72 may include a plurality of lenses, such as firstlens 72A and second lens 72B, and other focusing devices available tothose skilled in the art. A focus motor (not shown) may be employed tocontrol magnification of the data captured by the stereoscopicvisualization camera 12.

Referring to FIG. 2, the stereoscopic visualization camera 12 employs asecond light source 80 located in the head unit 18 and directed throughthe common objective lens set 72 along a second beam B2 onto the targetsite 16. The first beam B1 originating in the OCT module 14 isincorporated into the head unit 18 such that it falls upon some or allof the same portion of target site 16 as the second beam B2 falls. Inother words, the first beam B1 at least partially overlaps with thesecond beam B2 at the target site. This provides a number of technicaladvantages. The second portion 14B of the OCT module 14 connectsoptically to and from the head unit 18 via the sample light path L3. Thesecond portion 14B may be configured to communicate with the cameraprocessor 52 and controller C via the short-range network 56.

Referring to FIG. 2, the light arriving along the second beam B2 fromthe second light source 80 is reflected from the target site 16 alongthe first reflected light R1 back through the common objective lens set72 to the optical deflecting element 70 which reflects the light ofinterest along the second reflected light R2 toward the remainder of theoptical elements of the head unit 18. While in the example shown, theoptical deflecting element 70 is configured to bend the light path by 90degrees, it is understood that the angle may be varied based on theapplication at hand.

Referring to FIG. 2, a plurality of optical elements 84 may be employedto implement a channel (e.g. left or right) of a stereoscopic opticalpath and enable calibration of focus and zoom (magnification) duringproduction as well as dynamic zoom during use. The plurality of opticalelements 84 are configured to focus the second reflected light R2 ontoone or more sensors, such as first sensor 85 and second sensor 86, onefor each of the left view and right view making up a stereoscopic imageof the target site 16. The first sensor 85 and second sensor 86 areconfigured to sample the light field incident upon them and may includeCCD detectors or other types of detectors available to those skilled inthe art.

To enable the first beam B1 to fall upon some or all of the same targetsite 16 on which the second beam B2 does, the optical deflecting element70 may be coated or otherwise configured to selectively pass a givenpercentage of each incident wavelength of light from the first beam B1and selectively reflect the first reflected light R1. For example, inone embodiment, the useful spectral content of the first light source 40may reside in a Gaussian-type distribution of width from approximately740 nanometers of wavelength to approximately 930 nanometers ofwavelength, centered on a wavelength of approximately 840 nanometers.The useful spectrum of visible light for the first reflected light R1may be about 380 or 390 nanometers to about 700 nanometers. In thiscase, the optical deflecting element 70 may be coated to passwavelengths 740 to 930 nanometers as much as possible (typically 90% ormore) while reflecting wavelengths 380 to 700 nanometers as much aspossible (typically 90% or more).

While the second light source 80 of the stereoscopic visualizationcamera 12 is shown in the example in FIG. 2 as being non-coaxial to thefirst light source 40, it is understood that the location of the secondlight source 80 may be varied. For example, the position and orientationof the second light source 80 may be changed to allow co-axial lighting,such as by placing the second light source 80 behind the opticaldeflecting element 70 at location 88. The optical deflecting element 70may include a bandpass filter or partial pass regime to partially pass,and hence partially reflect, light in the visible region of thespectrum.

The common objective lens set 72 are configured to provide a variableworking distance W for the stereoscopic visualization camera 12. Theworking distance W may be referred to as the distance from a “center ofprojection” of an idealized camera model for the stereoscopicvisualization camera 12 to a reference plane where the target site 16 isin focus. Adjusting the common objective lens set 72 changes the workingdistance W of the head unit 18 and thus changes the effective opticallength of the sample arm 48.

Referring to FIG. 2, a working distance compensation member 90 may beemployed in the sample arm 48 to offset this change in order to ensurethat the reference arm 46 and the sample arm 48 have the same respectivenominal optical lengths. The working distance compensation member 90 mayinclude or more lenses, such as a liquid lens or other lens configuredto complement the common objective lens set 72. For example, ifadjusting the common objective lens set 72 increases the workingdistance by 10 millimeters, the working distance compensation member 90located in the sample arm 48 may be controlled (e.g. via the selector22) to similarly reduce the effective optical length of the sample arm48.

In another embodiment, the working distance compensation member 90 maybe located in the reference arm 46 and configured to match the workingdistance changes in the sample arm 48 by moving the reflecting surfaceof the reference arm 46 along the direction of travel of the referencelight path L2. For example, the working distance compensation member 90may include a micro-positioning stage to move the reflecting surface ofthe reference arm 46.

The OCT processor 50 and/or the controller C may be configured tocontrol the various components of the OCT module 14, including the firstlight source 40, the steering unit 64 and the working distancecompensation member 90. The working distance compensation member 90 maybe calibrated to match the changes in path length of the sample arm 48.The controller C may be configured to execute instructions to manageoverall operation of the OCT module 14. The instructions may be storedpermanently in the memory M or may be uploaded dynamically. The OCTprocessor 50 and/or the controller C may include sub-processors andother circuitry available to those skilled in the art to communicate andcontrol the various components of the OCT module 14.

The image stream from the stereoscopic visualization camera 12 may besent to the camera processor 52 and/or the controller C, which may beconfigured prepare the image stream for viewing. For example, thecontroller C may combine or interleave first and second video signalsfrom the stereoscopic visualization camera 12 to create a stereoscopicsignal. The controller C may be configured to store video and/orstereoscopic video signals into a video file and stored to memory M. Toview the stereoscopic display, a user may wear special glasses that workin conjunction with the first and displays 32, 34 to show the left viewto the user's left eye and the right view to the user's right eye.

The controller C of FIG. 1 is specifically programmed to execute theblocks of the method 100 (as discussed in detail below with respect toFIG. 3) and may include or otherwise have access to executable programsor information downloaded from remote sources. Referring to FIG. 1, thecontroller C may be configured to communicate with a remote server 92and/or a cloud unit 94, via a long-range network 96. The remote server92 may be a private or public source of information maintained by anorganization, such as for example, a research institute, a company, auniversity and/or a hospital. The cloud unit 94 may include one or moreservers hosted on the Internet to store, manage, and process data. Thelong-range network 96 may be a Wireless Local Area Network (LAN) whichlinks multiple devices using a wireless distribution method, a WirelessMetropolitan Area Networks (MAN) which connects several wireless LANs ora Wireless Wide Area Network (WAN) which covers large areas such asneighboring towns and cities. Other types of connections may beemployed.

The controller C may be configured to receive and transmit wirelesscommunication to the remote server 92 through a mobile application 98,shown in FIG. 1. The mobile application 98 may in communication with thecontroller C via the short-range network 56 such that it has access tothe data in the controller C. In one example, the mobile application 98is physically connected (e.g. wired) to the controller C. In anotherexample, the mobile application 98 is embedded in the controller C. Thecircuitry and components of a remote server 92 and mobile application 98(“apps”) available to those skilled in the art may be employed.

II. Example Method or Implementation

Referring now to FIG. 3, a flow chart is shown of an exampleimplementation or method 100 of the system 10. It is understood that themethod 100 need not be applied in the specific order recited herein andsome blocks may be omitted. The memory M can store controller-executableinstruction sets, and the processor P can execute thecontroller-executable instruction sets stored in the memory M. Themethod 100 enables determination of a shared composite view based on thestereoscopic visualization camera 12 and optical coherence tomography(OCT) module 14.

Per blocks 102 and 104 of FIG. 3, the OCT module 14 is calibrated and afirst set of volumetric data is obtained of the target site 16, via theOCT module 14. As will be described in detail below, calibrating the OCTmodule 14 includes calibration along an axial direction A, firsttransverse direction T1 and second transverse direction T2 (see FIGS.4A-C). Per blocks 106 and 108 of FIG. 3, the stereoscopic visualizationcamera 12 is calibrated and a second set of volumetric data (i.e. cameradata in three dimensions) from the stereoscopic visualization camera 12is obtained.

Per block 110, the controller C is configured to register the first setof volumetric data from the OCT module with the second set of volumetricdata from the stereoscopic visualization camera 12 to create a third setof registered volumetric data. Disparity mapping may be employed suchthat the respective output images of the stereoscopic visualizationcamera 12 are positioned in the same space as the respective outputimages of the OCT module 14. The disparity map includes the estimatedpixel difference or motion between a pair of stereo images. Calibrationand registration are described in detail below with respect to FIGS.6-18.

Per block 112, the third set of registered volumetric data may berendered, via the VR module 51 (see FIG. 1), to a first region to obtaina two-dimensional OCT view. Per block 114, the second set of volumetricdata from the stereoscopic visualization camera 12 is rendered, via theVR module 51, to a second region to obtain a live two-dimensionalstereoscopic view. Volume rendering refers to a three-dimensional volumereconstruction method that allows every voxel in a volumetric data tocontribute to the reconstructed image. Stated differently, volumerendering is a set of techniques used to display a 2D projection of a 3Ddiscretely sampled data set. The first and second regions may comprise aphysical memory storage unit, such as a buffer, used to temporarilystore data.

Per block 116, the first region and the second region are overlaid toobtain a shared composite view of the target site, which may be shown onat least one of the first and second displays 32, 34. An example of ashared composite view 300 of an eye E is shown in FIG. 5 and describedbelow. Per block 118, the controller C may be configured to extractstructural features from the shared composite view 300. The first set ofvolumetric data may be configured to be updated at a first frequency andthe second set of volumetric data may be configured to be updated at asecond frequency. In one example, the updating of the first set ofvolumetric data and the second set of volumetric data is notsynchronized. Stated differently, the stereoscopic visualization camera12 and the OCT module 14 may define a respective latency.

To mitigate this, the controller C may include a first set of imagebuffers configured to selectively delay the display of thetwo-dimensional OCT view in order to match the respective latency of thestereoscopic visualization camera 12. The controller C may include asecond set of image buffers configured to do the opposite, andselectively delay the display of the two-dimensional stereoscopic viewto match the respective latency of the OCT module 14.

III. System Operation

Referring now to FIGS. 4A, 4B, 4C and 4D, example scanning regions forthe OCT module 14 are shown. FIGS. 4A and 4B are schematic fragmentaryperspective views while FIGS. 4C and 4D are schematic fragmentary topviews of example scanning patterns. Referring to FIG. 4A, a single scandirected at the spot scan 60 of the target site 16 results in a depthscan 202 of the structure of the physical sample into which the firstbeam B1 is directed, along the incident direction. Referring to FIG. 4A,the depth scan 202 may be referred to as an “A-scan” and is configuredto scan to a detected depth 204 along an axial direction A. The axialdirection A which is the travel direction of the first light source 40in the example shown in FIG. 2.

The first beam B1 of FIG. 2 may be moved in a continual manner about thetarget site 16 using the steering unit 64, thereby enabling a seconddepth scan 206, a third depth scan 208, a fourth depth scan 210 and afifth depth scan 212 along a first transverse scan range 214, forexample. Such a line of A-scans may be referred to as a B-scan or rowscan 216.

Referring to FIG. 4C, by steering the optical path appropriately alongthe first transverse scan range 214, then performing a “step-and-repeat”path steer along the raster pattern 218 to repeat the cycle at astarting point 220 and subsequent lines, a grid of depth scans may betraced out along the target site 16, along the first transverse scanrange 214 and a second transverse scan range 222. Referring to FIGS. 4Aand 4B, this results in a three-dimensional sampled volume havingboundaries 224, which may have the shape of a cuboid. The steering unit64 may be moved continually along the raster pattern 218. The boundaries224 of the sampled volume may be determined during OCT calibration asdescribed below.

Referring to FIG. 4A, the detected depth 204 or penetration depth for adepth scan 202 is dependent on many factors, including the spectrum ofthe first light source 40 at the starting point 220, the opticalcharacteristics of the starting point 220 over the spectrum and thespectral resolution of the detector 44. Similarly, the starting point220 may be drawn as the near extent of the dataspace of the OCT module14; the first detected point depends on the shape and characteristics ofthe target site 16. Reflection points may appear as “bright” pixels inthe line-scan camera data. For example, if the possible pixel values arein the range 0-255, non-reflection points might have a value of 30 orless, while bright reflection points might have a value of 150 orgreater. For example, with materials such as human skin or the humaneye, which are transparent or semi-transparent at the wavelengthscontained in the first beam B1, the depth scan 202 may penetrate somemillimeters into the material.

The movement of the first beam B1 by the steering unit 64 along with theprocessing of each A-scan (e.g. second depth scan 206, a third depthscan 208, a fourth depth scan 210 and a fifth depth scan 212) may besynchronized with the rest of the system 10 by the controller C and/orthe OCT processor 50, such that the downstream processes may reassemblethe scans in the same order and relative location during thereconstruction process. The absolute location may be determined via thecalibration and registration methods described below.

In one embodiment, the steering unit 64 includes a multi-axisgalvanometer or a set (for example, a pair) of single-axisgalvanometers. A single-axis galvanometer is a small lightweight mirrorthat can rock back and forth on an axis under electrical control,thereby enabling changing the reflection direction of the path of lightreflected in the mirror, but only about one axis. This enables B-scansin one transverse direction. A multi-axis galvanometer may be consideredto be two single-axis galvanometers taken together. One steers the lightalong the B-scan direction in one transverse direction while the othersteers the light along the B-scan direction in the other transversedirection. In other embodiments the steering device may be implementedas a digital micro-mirror which is typically a smaller and morelightweight version of a galvanometer but manufactured using siliconcomputer chip production processes. When the steering unit 64 operatesusing continuous movement, some blurring may occur of the data obtainedby the OCT module 14 due to movement of the steering unit 64 occurringduring sampling. This may be mitigated by including usingstepper-motor-driven mirrors instead of galvanometers in the steeringunit 64. The steering unit 64 may be selected based on the applicationat hand and the speed, weight, resolution, accuracy and repeatabilitydesired.

Resolving Power and Depth of the OCT Module

The sampling resolution of the system 10 is a function of the resolutionin the axial direction A (the direction of the A-scan), the diameter ofa single A-scan and the separation of adjacent A-scans in each of thetwo remaining directions, the first transverse direction T1 and thesecond transverse direction T2. In one example, two different axialresolutions are possible: a first axial resolution for “high-depth,lower resolution” mode and a second axial resolution for “low-depth,higher resolution” mode. In one example, the first and second axialresolutions are approximately 20 micrometers (20 μm) and 2 micrometers(2 μm), respectively. The different axial resolutions may be implementedin this embodiment using two different light sources: a higher bandwidthlight source for “low-depth, higher resolution” and a lower bandwidthlight source for “high-depth, lower resolution”. For optimal performanceeach respective light source may be mated with its owndetector/spectrometer to best take advantage of the resultant bandwidth.Switching between the light source/spectrometer pairings may be achievedusing an optical switch (not shown).

Referring to FIG. 4C, the diameter of the spot scan 60 may berepresented by a first set of dimensions 240 and 242 and are related tofactors such as the structure of the first light source 40 and thesubsequent optical path encountered by the first beam B1. In oneexample, the first set of dimensions 240 and 242 are equal and in therange of approximately 15 μm to 80 μm. In other examples, first set ofdimensions 240 and 242 are not equal. Referring to FIG. 4C, theseparation of adjacent ones of the A-scans or depth scans 202 may berepresented by a second set of dimensions 244 and 246, which may beequal and in the range of approximately 15 μm to 80 μm. The separationis related to factors such as the optical distance from the steeringunit 64 to the target site 16 and the resolution of the steering unit64.

As noted above, the steering unit 64 may include plurality of steeringmembers, such as first steering member 65, second steering member 66 andthird steering member 68. The resolution of the steering members affectsthe separation. The span of the first and second transverse scan ranges214, 216 are related to the properties of the steering unit 64. Each ofthe values of the resolution in the axial direction A and the first andsecond transverse directions T1, T2, the axial span or detected depth204, and the spans of the first and second transverse scan range 214 and222 may be respectively determined during an OCT calibration process asdescribed below.

Other Scan Patterns

Referring to FIG. 4D, other scan patterns are possible by changing therelative timing of the steering unit 64. For example, a radial scanpattern 250 may be implemented wherein a circular region 252 is sampledstarting for example at scan start point 254, proceeding along radialpath 256, performing a “step” along path 258 and “repeat” of (or inother embodiments, continuous travel along) the sample path 260. Asimplified, faster version of this scan may include a “two B-scan”process with a single pair of scans oriented perpendicular to each otherin transverse directions, which enables faster determination ofmovements of the target site 16 up to a speed such that the movement canbe detected in these limited scans.

Referring to FIG. 4D, the radial scan pattern 250 may be employed toobtain a plurality of radial B-scans along paths 260, 262, 264, 266 and268, and sample a cylindrical-type volume. Such a sampling pattern isadvantageous, for example, when working with the surfaces of the humaneye where it may be desirable to measure two-dimensional curvatures ofvarious surfaces. The curvature is seen to lie along the sample lines,especially if the center of the sample volume is aligned with the centerof curvature of the surface, as is achieved approximately using theraster pattern 218 and extracting the mean highest point, eitherdirectly from samples or using curve fitting among several-to-manysamples. Referring to FIG. 4C, further improvements for such a radialscan pattern 250 include filling the cylindrical sample space by addingA-scans along connected line segments 270, which include at least twoconnected line segments separated by an angle.

Referring to FIG. 5, an example of a shared composite view 300 of an eyeE is shown on the first display 32 (or second display 34 of FIG. 1).Also shown in FIG. 5 are the pupil 302 and iris 306 of the eye E. Thefront and rear surfaces of the eye E captured by the OCT module 14 maybe separately extracted, and a respective topographic map generated foreach such surface and overlaid with the view from the 2D visualizationmodules V1, V2 of the stereoscopic visualization camera 12, for example,using different patterns to represent relative depth. Referring to FIG.5, a live stereoscopic image L of the eye E is overlaid with atopographic outline 312. The topographic outline 312 may be representedby a plurality of topographic levels, such as a first topographic level314 and a second topographic level 316. The number of possibletopographic levels is related to the axial resolution of the OCT module14. The controller C may be configured to add at least one annotation320 (dashed circle in FIG. 5) showing a boundary of the two-dimensionalOCT view over the shared composite view 300. A second annotation 330 mayindicate a landmark of the eye E. The controller C may be configured tomaintain the relative position of the second annotation 330 on the firstdisplay 32 relative to other features of the eye E.

IV. System Calibration

The system 10 is configured to fuse information provided by two imagingmodalities: the stereoscopic visualization camera 12 and the OCT module14. The stereoscopic visualization camera 12 includes two 2Dvisualization modules V1, V2, for example, two cameras generallyproviding a left view and a right view of a stereoscopic image. Toenable accurate and clinically useful fusion of image space informationof the stereoscopic visualization camera 12 and the OCT module 14, thesystem 10 is configured to represent to the user the captures made byeach of these two imaging modalities respectively as if they representthe same three-dimensional object, in all dimensions. Data from specificportions of the target site 16, from each modality, must be displayed inthe same location or otherwise imply co-location. Additionally, thesystem 10 is configured to match the size and orientation throughout theimage, and for images that represent multiple dimensions, across alldimensions (referred herein as matching perspective). Further, inembodiments comprising fused display of both stereoscopic visualizationand stereoscopic OCT images, a left and a right view from each modalityis required in combination, to comprise the image set. For optimalaccuracy and visual comfort, the first and second views of each modalitymust appear to have identical stereoscopic parameters, such asrespective intraocular distance, scale, perspective and orientation.

To do this, the captures made by each of these two imaging modalitiesrespectively are converted to a common space in a calibration processand aligned where appropriate in position, orientation and size, in aregistration process. Additionally, since the presentation to the userof such an image fusion typically relies on display devices such as aflat-panel display, the method of “moving to display space,” for exampleby creating a two-dimensional representation, must result in therelevant parameters being represented in a matched way in the finalrepresentation for each imaging modality.

Each of the modalities used here capture different representations orproperties of the target site 16. The stereoscopic visualization camera12 captures data related to visible light while the OCT module 14captures a set of intensities related to the position of structures onand inside the target site 16. For example, the nearly transparentanterior and posterior surfaces of the cornea are imaged by the OCTmodule 14 but are often difficult to clearly see with the stereoscopicvisualization camera 12. The incoming data is arranged in each caseaccording to the physical parameters of position, orientation and sizeof the scene objects, with the two modalities sharing these physicalparameters. Use of the datasets concurrently is done by converting themto a common base dimension (such as millimeters), and registering theirimages by aligning them in position, orientation and scale relative toeach other and/or to a known reference point. In other words, therespective image space representations of a given scene or portion of ascene will appear to be the same size, in the same orientation, at thesame position, to within some tolerance.

Calibrating the OCT Module

Calibrating the OCT module 14 includes axial calibration and transversecalibration. Additionally, compensation must be made for the index ofrefraction of materials that the first beam B1 passes through in thetarget site 16. Environmental variation may be compensated for byperforming the calibrations in various environmental conditions andinterpolating and extrapolating the results to the current conditions.For example, the axial calibration may be performed at multipletemperatures across the expected operating range of the head unit 18 andthe results as a function of temperature are fitted to a calibrationcurve of various complexity. Then the operating temperature of thecomponents of the head unit 18 may be measured at the time of a surgeryand the calibration curve may be used to determine the actual axialcalibration for the current surgery.

Axial Calibration of the OCT Module

Axial calibration is calibration of the distances along the first beamB1 (see FIG. 2) such that pixels in the OCT data along the depthdirection or axial direction A are converted to physical units e.g.millimeters (one-thousands of a meter) or micrometers (one-thousandthsof a millimeter). FIG. 6A is schematic illustration of an axialcalibration set-up 400 at the reference arm 46 of the OCT module 14.FIG. 6A shows the beam splitter 42 and the reference arm reflectingdevice 62. The reference arm reflecting device 62 may be a flatfirst-surface mirror. In other embodiments, the reference arm reflectingdevice 62 may be a corner cube or other optical structure capable ofreflecting a beam along a controlled direction with minimal loss.Referring to FIG. 6A, the axial calibration set-up 400 includes acalibrated linear motion mechanism or linear actuator 404 having acontrol and readout module 406 in the reference arm 46.

FIGS. 6B, 6C and 6D respectively illustrate a top view, a side view anda front view of a calibration device 450 for axial calibration of theOCT module 14. The calibration device 450 may be made with a nominallyflat surface which is reflective to some degree. The calibration device450 is placed in a fixed position at the scene location, i.e. the targetsite 16, which becomes the nominal location for all other measurementsand which flat, reflective surface reflects some of the first beam B1back toward the beam splitter 42. To facilitate relating the axialcalibration information of the OCT module 14 to the focal plane F of thestereoscopic visualization camera 12, the calibration device 450 is alsotextured to assist with focusing the stereoscopic visualization camera12 onto the surface. The texture is non-periodic and may be one or moreof the following: inherent in the material; printed and/or affixed; andprojected.

Referring to FIGS. 6B to 6D, the calibration device 450 has a firstdimension 452, a second dimension 454 and a third dimension 456. Thefirst dimension 452 or thickness dimension of the calibration device 450is configured to be much less than the other two dimensions (e.g. 454and 456) to avoid negative effects of axial imaging in that dimension,such as shadowing. The stereoscopic visualization camera 12 is focusedonto the surface of the calibration device 450 and the value of theposition of the linear actuator 404 is recorded.

Referring to FIGS. 6B to 6D, the calibration device 450 may be made intoan irregular shape with a first segment 458 and a second segment 460,each with differing structural properties and differing sizes, todetermine orientation. An automated algorithm available to those skilledin the art may be employed to separate the data from the first segment458 and the second segment 460.

Referring now to FIG. 6A, the linear actuator 404 is first set to astarting position 410 which becomes the nominal location of thereference arm mirror 402. A first measurement is made of this setup withthe surface of the calibration device 450 and reference mirror 402 asdescribed. Referring to FIG. 7A, an output image 470 (B scan) of thesurface is created and a first line 472 of “bright pixels” is observedrepresenting a top surface of the calibration device 450. The locationof the first line 472 of bright pixels along the axial direction A inthe data is recorded. FIG. 7B illustrates a pixelated output image 480.Referring to FIG. 7B, in the example shown, the first line 472 is seento occur at about 20% of the image height H (shown in FIG. 7A).

Referring to FIGS. 7A and 7B, a second line 474 of less-bright pixels isalso seen and represents the rear surface of the calibration device 450.Such a line may be used as a secondary confirmation of measurement andto calibrate the effect of index of refraction of the material of thecalibration device 450. Alternatively, once the axial calibration of theOCT module 14 is complete, the thickness of the calibration device 450is measurable by counting the separation in pixels of the second line474 from the first line 472. The vertical position in the OCT datacorresponds to a focus distance of the stereoscopic visualization camera12 represented by the setting of the linear actuator 404 at thereference arm mirror 402, and is recorded with the focus setting as adata pair such that each value may be retrieved by looking up the pairusing the other value. The data pair set may be employed to match OCTdata and the image data from the stereoscopic visualization camera 12 bykeeping or maintaining the setting of the linear actuator 404 at itscurrent value during operation. The linear actuator 404 with readout andcontrol 406 is then moved an amount smaller than the expected range(e.g. detected depth 204 along the axial direction A as shown in FIG.4A) of a given A-scan. This amount is typically one to three orders ofmagnitude less than the expected range. The focus of the stereoscopicvisualization camera 12 is not changed. The corresponding change in“bright pixel” position in the data corresponds to the change inposition of the precision linear motion mechanism for this focus settingof the stereoscopic visualization camera 12. The focus setting isincluded because the first beam B1 of the sample arm 48 passes throughand is therefore affected by the objective lens set 72 (see FIG. 2) ofthe stereoscopic visualization camera 12. Thus for example if the linearactuator 404 is moved 1 millimeter and the “bright pixel” moves 100pixels in the data then each pixel in the data is calculated torepresent an axial length of 1 millimeter divided by 100 which is 10micrometers (10 μm.) This value is added to the data pair entry for thisfocus setting. By continuing this process and thus moving the linearactuator 404 along its range, the axial range of the OCT module relativeto the focal plane of the stereoscopic visualization camera 12 at thisfocus setting and actuator setting is found either directly or byextrapolation.

The OCT axial calibration data may be adjusted so that the focus settingappears at the top of the image (at 0% image height). For example, if itis desired to image only below a surface on which the stereoscopicvisualization camera 12 is focused, then the location of the nominalposition of the focused surface in the OCT data may be moved toward thetop of the OCT data range. Controlling the axial offset of the OCT dataaxial range may be implemented as follows. The stereoscopicvisualization camera 12 is focused on the axial calibration device 450and the linear actuator 404 (see FIG. 6A) is adjusted until the firstline 472 bright line of pixels appears at a desired first verticallocation 476 (see FIG. 7B), in the data, e.g. at pixel row 0. The secondline 474 appears at a second vertical location 478 (see FIG. 7B). Thisoffset position for the reference arm 46 may optionally be used as themounting position for the permanent reference arm mirror mount if used,for example, when the working distance compensation member 90 isimplemented in the sample arm 48. The axial calibration may be performedusing just a single A-scan. However, the B-scan offers the opportunityto detect the position of the axial calibration device 450 in onedimension as well as its tilt in one dimension relative to first beam B1of the OCT module 14. Performing a complete three-dimensional volumetricsampling provides information about the other two dimensions.Additionally, such tilt and position detection assists in registrationof the OCT space to the respective camera spaces of the 2D visualizationmodules V1, V2 of the stereoscopic visualization camera 12.

Transverse Calibration of the OCT Module

Calibration in each of the first and second transverse dimensions T1, T2of the OCT scan space is achieved by viewing a patterned target with theOCT module 14. An example of a patterned target 500 is shown in FIG. 8A.The patterned target 500 has multiple bar patterns 502 with knownrespective widths 504, heights 506 and respective gaps 508 at multiplescales. The patterned target 500 may be printed or otherwise made toappear on a predominantly planar substrate. The thickness of thepatterned target 500 is configured to be much smaller in magnitude thanthe other two dimensions. This reduces potential negative effects ofimaging the third dimension, such as shadowing, from impacting thetransverse calibrations. For example, the patterned target 500 may beprinted e.g. via photolithography with chromium on a glass substrate,with a largest positive target width of 5 mm and a thickness of about 1um or 10 um.

Referring to FIG. 8B, another example of a calibration target 520 isshown. The calibration target 520 may include multiple respectivecalibration sub-targets 522 having differing sizes of width, height andseparation. Additionally, the calibration target 520 may includemultiple varying scales 524, 526, 528. It is understood that otherpatterns and/or targets available to those skilled in the art may beemployed.

Referring now to FIG. 9A, a schematic perspective view of an apparatus600 is shown, with a plurality of calibration devices 602 for transversecalibration of the OCT module 14. The plurality of calibration devices602 (which may include the patterned target 500, calibration target 520and/or other targets), is placed in the field of view and imaged by theOCT module 14. FIGS. 9B and 9C illustrate example data received from theapparatus 600 of FIG. 9A. FIGS. 9B and 9C illustrate a first outputimage 650 and a second output image 670, respectively. The second outputimage 670 shows the first output image 650 with pixel divisions. It isunderstood that the FIGS. are not to scale.

Referring to FIG. 9C, the A-scan direction 652 is vertical in the firstand second output images 650, 670 and the B-scan direction 654 ishorizontal. The predominant plane of the calibration devices 602 isassumed to be positioned orthogonally to the first beam B1. The topsurface 634 (see FIG. 9A) is detected as a horizontal line 636. Forfurther simplicity, the rear surface is not shown in these OCT images.The difference between a positive (e.g. chromium pattern line) portion637 and a negative (e.g. empty or blank) portion 638 of the calibrationdevices 602 is discernible in the horizontal line 636 in the OCT data asdifferences in thickness and/or intensity.

From the first and second output images 650, 670, the pixel width 640 inpixels of the positive target lines may be read as approximately 4pixels wide. Similarly, for the pixel width 642 of the negative targetlines. With knowledge of the respective physical widths 644 of thecalibration devices 602 and separation 646, the conversion is enabledfrom pixels in the transverse direction of the OCT data to physicaldistance in the real world. For example, if the OCT module 14 images thepatterned target 500 of FIG. 8A, where the positive region is 1 mm wideand the negative region is 1 mm wide, and in the data it is measuredmanually or detected by the algorithm that the width in pixels of thepositive portion of the patterned target 500 is 100 pixels, then thetransverse resolution is calculated to be 1 mm divided by 100 pixelswhich is 10 um per pixel. The negative portions of the patterned target500 may be used in some embodiments along with the positive portions toreduce the effects of measurement bias and to increase the overallnumber of measurement samples which increases accuracy by reducingmeasurement noise.

The transverse calibration method described above enables thedetermination of the transverse resolution of the steering unit 64 (seeFIG. 2) of the OCT module 14. The change in movement control settings ofthe steering unit 64 between adjacent A-scans may be obtained bysub-components in the OCT processor 50 and/or controller C. Assuming thechange amount is constant and the OCT beam angle does not changeappreciably between A-scans, the conversion from change amount tophysical distance may be obtained by dividing the change amount by thephysical distance between pixels in the transverse direction ofinterest. Change amounts in the other transverse dimension may be foundsimilarly. Alternatively, the change amount may be taken over manypixels (for example 10 or 100) at once and dividing it by the physicaldistance those many pixels are found to cover.

Fan Distortion

Referring to FIG. 2, the first beam B1 from the OCT module 14 isgenerally non-parallel across A-scans. This results from a narrow sourceand narrow beam being directed across a volume significantly wider thanthe source and beam and is referred to herein as fan distortion.Refinements may be made to account for fan distortion by comparing pixelpatterns of a calibration device among multiple scans made with thecalibration device varying in axial distance from the OCT module 14. Thechanges in axial distance are known from the axial calibration step, forexample, by being made using a calibrated linear actuator 404 such asshown in FIG. 8A. The separation of positive and negative target regionsin the OCT image data may be automated using an image processingalgorithm for detecting edges available to those skilled in the art.

Transverse Offset

Similar to how axial offset is optionally adjusted to control where the“start” of the OCT data occurs in the axial dimension in physical space,the transverse scan area may be adjusted to control where the “start” or“origin” of the OCT data occurs in both of the transverse dimensions(e.g. the first and second transverse dimensions T1, T2 in FIG. 4A) inphysical space. Controlling this offset is useful so that the spot scan60 of the OCT module 14 can be moved about a given scene, as well as toset its origin in a known location either relatively or absolutely to areference point or to the OCT module 14. For example, the upper leftcorner of the focal plane of one of the two 2D visualization modules V1,V2 of the stereoscopic visualization camera 12 may be used as such areference point. The measurement of the transverse offset may be foundusing the above transverse calibration method and recorded for lateruse.

Locating OCT Space Relative to the Target Space

FIG. 10A is a schematic perspective view of a set-up 700 for determiningthe location and orientation of respective data space of the OCT modulerelative to a calibration device defining a target space. As noted abovewith respect to FIG. 1, the system 10 may include a robotic arm 24operatively connected to and configured to selectively move the headunit 18. The target space origin 702 (and orientation) of the targetspace may be obtained relative to the head unit 18 (housing respectiveportions of the stereoscopic visualization camera 12 and OCT module 14)by mounting the head unit 18 on the robotic arm 24 locating the targetat a known location relative to a robot base (also known as the robotreference frame). The joint angles and link lengths of the robotic arm24 may be obtained or are known by measurement e.g. by reading robotsensors. In another embodiment, referring to FIG. 1, a mounting stand 25with respective mounts for the head unit 18 and the calibration target520 may be employed, the respective mounts having known relativeposition and orientation.

Referring to FIG. 8B, the calibration target 520 includes multiplerespective calibration sub-targets 522 having differing sizes of width,height and separation. Referring to FIG. 10A, the OCT space origin 704relative to the target space may be found in similar ways by using arespective calibration sub-target 522 larger than the OCT space in somedimensions versus a respective calibration sub-target 522 smaller thanthe OCT space. In one example, a respective calibration sub-target 522is employed that is large enough to fill or overfill the two transversedimensions T1, T2 of the OCT space. The respective calibrationsub-target 522 has features of known dimension, of known location andorientation relative to the robot reference frame and provides a meansto differentiate features from each other or count them. When thisrespective calibration sub-target 522 is placed in the scene such thatit overfills those two transverse OCT dimensions T1, T2, then the OCTspace origin 704 is defined to be located at the point on the respectivecalibration sub-target 522, where the X=0, Y=0 pixel of the OCT data isimaged. The Z=0 location is adjusted for example to the calibrationtarget surface using the axial offset calibration as described above.Alternatively, using a respective calibration sub-target 522 that doesnot fill the two transverse dimensions of the OCT space but which isplanar, extrapolation 756 (see FIG. 10B) with a first group 716 ofmultiple A-scans and a second group 720 of multiple B-scans may be usedbeyond the extents of the respective calibration sub-target 522 tocalculate the extents of the OCT space in the two transverse dimensionsT1, T2 relative to the respective calibration sub-target 522.

Orienting the OCT Space Relative to the Target Space

Referring to FIG. 10A, a real-world space including a calibration target520 bounded by respective OCT extents 710 is shown along with thearrangement of a single A-scan 712 along an axial direction A, as wellas a first group 716 of multiple A-scans comprising a B-scan 718 in thefirst transverse direction T1, as well as a second group 720 of multipleB-scans in the second transverse direction T2. The OCT space origin 704relative to the target space origin 702 may be determined by fitting aline to the first surface of the calibration target 520 in each of thethree orthogonal views O1, O2 and O3 of the OCT data.

FIG. 10B is a schematic front transverse view of example data receivedfrom the set-up 700. FIG. 10B shows the upper detected edge 752 (andlower detected edge 753) of the calibration target 520 running parallelto the defined X-axis 754 of the target space. The extrapolation 756 isshown intersecting with the X-axis 758 of the OCT space. The angle 760of this intersection is the angle that the respective X-axis of thecalibration target 520 is rotated about the Y-axis 762 of the OCT space.The “inverse” of this relation is that the OCT space's Y-axis 762 isrotated about the X-axis of the calibration target 520 by the negativeof the angle just found.

The relative rotations in the other dimensions may be found similarly.The origin of the calibration target 520 in the OCT space may be foundsimilarly, with the X and Z positions being obtained as an x-axis offset770 and a z-axis offset 772, respectively, using the front transverseview (see FIG. 10B). The y-axis offset may be obtained from a differentview. The location and orientation of the OCT space relative to thetarget space are taken together and referred to herein as the firsttransformation 774 of the OCT space relative to the target space.

Mathematically this information may be encoded in a transformationmatrix which is read “backwards” as “the transformation from OCT spaceto target space.” The inverse of this transformation matrix is taken tofind that “the transformation from target space to OCT space” is theinverse of “the transformation from OCT space to target space.” Thecontroller C and/or the camera processor 52 and/or OCT processor 50 mayinclude utilities, mathematical programs and other facilities availableto those skilled in the art to compute this inverse as well as tomultiply such matrices together.

Scene Capture in 2D Visualization Modules V1, V2 of StereoscopicVisualization Camera

FIGS. 11A and 11B are schematic perspective views of a set-up 800showing location and orientation of the calibration target 520 relativeto one of two monoscopic views of the stereoscopic visualization camera12. Viewing a given scene with a given imaging modality results in a“capture” of the scene view into the imaging modality's image space. Thescene capture may be modeled in a number of ways, including with the useof a pinhole camera model. At a given zoom and working distance setting,the 2D visualization modules V1, V2 of the stereoscopic visualizationcamera 12 may be modeled as a simple pinhole camera wherein light from ascene (such as calibration target 520) travels through a single“pinhole” also known as the center of projection (COP) 802 and isprojected such as a virtual image 804 onto an image sensor 806. Thisresults in a two-dimensional image 820 (see FIG. 11B) of the calibrationtarget 520 for each 2D visualization module V1, V2. For monochromaticsensors and image pixels, the image is a two-dimensional grid of pixels,the values of which are related to the light coming in from the part ofthe calibration target 520 which falls onto the correspondinglight-sensing elements in the sensor. For color pictures, a red, greenand blue filter may be employed over sensor pixels and a recombiningalgorithm used to combine the colors into color output pixels.

Referring now to FIG. 12A, the location and orientation of therespective data space of the OCT module 14 relative to one of the twomonoscopic views of the stereoscopic visualization camera 12 is shown.In the set-up 860 of FIG. 12A, the origin 812 of the coordinate systemor respective space of the 2D visualization module is taken to belocated at its center of projection 802. The coordinate system of anobject may be referred to as the “space” of the object so that thecoordinate system of the 2D visualization module may be referred to asthe “COP space.”

Referring to FIG. 11B, the image 820 may be considered as located in theCOP space, for example, with projected axes located at position 822 andoriented as shown at axes 824. The orientation may be chosen to eitherobey conventions in the field of art (for example with the camerapointing down the negative Z-axis and the “right” and “up” axes of thecamera being aligned along the X- and Y-axes respectively). The imagemay conceptually be considered located on the opposite side of the COP802 which can flip and/or negate some axes. In some embodiments, thecoordinate systems may have different origins and/or orientations and/orhandedness where handedness is the relation of the X, Y, and Z axesrelative to each other. For monoscopic display, the image produced maybe drawn directly to the first and second displays 32, 34. Stereoscopicdisplay entails concurrent display of (for example interleaving) theimage data from each 2D visualization module, and steering the displaylight from one 2D visualization module to one of the user's eyes andsteering the display light from the other 2D visualization module to theother of the user's eyes. Such steering may be done by usinglight-polarizing films of varying properties on both the display 32 andthe user's glasses.

Scene Capture in the OCT Module

Capturing a single A-scan or depth scan 202 of a view of the scene ortarget site 16 into the image space of the OCT module results in asingle line of data. This may be somewhat equivalent conceptually to asingle column of pixels in an image captured by one of the 2Dvisualization modules V1, V2 of the stereoscopic visualization camera12. Using the steering unit 64 to move the first beam B1 in a line, forexample along axial direction A, to capture multiple A-scans results ina single B-scan which is stored in the OCT image data space as a singletwo-dimensional image, such as the output image 470 shown in FIG. 7A.Using the steering unit 64 to move the first beam B1 in a raster-typepath such as raster pattern 218 (see FIG. 4C) results in multipleB-scans which are stored in the OCT image data space as multipletwo-dimensional images such as the output image 470 of FIG. 7A. Theyrepresent a three-dimensional sampling of the three-dimensional scenespace covered by the OCT space boundaries 224. The resultant data isthree-dimensional when OCT data is organized as a sequence oftwo-dimensional B-scan images with the A-scans comprising either thevertical columns or the horizontal rows of the B-scan images, dependingon configuration. Computer graphics technology may be employed withstorage mechanisms, such as a three-dimensional texture as well as meansto sample such data, from any three-dimensional location within thedata's span. Such sampling includes algorithms to interpolate valueswhen the sample position does not coincide with the original dataposition.

Displaying three-dimensional data may be achieved using the VR module51, which results in two-dimensional images suitable for viewing on atwo-dimensional display. Referring to FIG. 12A, the virtual center ofprojection of the VR module 51 (characterizing a virtual volume renderercamera) may be positioned virtually in the OCT space based on parameterssuch as the third transformation 862 (represented as a matrix) shown inFIG. 12A, determined during calibration and registration of the OCTmodule 14 and the stereoscopic visualization camera 12.

The VR module 51 of FIG. 2 includes a volume renderer program togenerate the two 2D images from the OCT data set. In the VR module 51,rays are cast from pixel on “virtual sensor” toward data, the rays beingthe projected image. A stepping function is used to “grow” the raylonger and longer from the COP toward and through the OCT data and thethree-dimensional texture in which the OCT data is stored is sampled ateach step. When the ray is outside the OCT data, or the value of thedata is zero at the ray sample point, a value of zero is considered toexist at that ray position. The final value of the pixel at the virtualimage is taken as a function of the values encountered at these steps.In one embodiment such a function merely takes the maximum valueencountered along the ray. Other functions accumulate the values andsegment the data by thresholding the result and present differentgroupings as different colors for example, to enable differentiation ofdifferent tissue types in a human brain for example. Other embodimentsmake an estimate of the orientation of the surface at any given point inthe data, which orientation is then used along with a lighting model tocreate the effect of a lighted surface. Resolution may be controlled bythe number of pixels in the virtual image plane as well as the number ofsteps per ray.

Calibration of the Stereoscopic Visualization Camera

Optical calibration characterizes the intrinsic optical parameters of agiven 2D visualization module having some known parameters such as pixeldimensions and other unknown parameters such as focal length and opticaldistortion. This enables knowledge of how a real-world structure ofknown dimensions will be represented in the image space of the 2Dvisualization module and hence also enables the converse action towithin a scaling parameter: determining the dimensions of a real-worldstructure (to within a scaling parameter) using its image spacerepresentation in a given optically calibrated 2D visualization module.Optical calibration can also reveal extrinsic optical parameters whichrelate the position and orientation in the scene of a calibration objectof known dimensions relative to the camera. Further, calibration of thestereoscopic visualization camera 12 determines the relationship betweentwo 2D visualization modules V1, V2 of similar optical propertiespositioned and oriented near each other and enables determination ofabsolute values in a given set of units (e.g. millimeters) ofmeasurements of arbitrary structures in the scene (the dimensions,positions and orientations of which do not need to be known beforehand),such as their absolute size and their position and orientation relativeto some reference point on the stereoscopic visualization camera 12.

Pinhole Camera Model and Calibration

The 2D visualization modules V1, V2, at a given zoom and focus settingmay be respectively modeled as a simple pinhole camera. Each of the two2D visualization modules V1, V2 in the stereoscopic visualization camera12 is first modeled as such separately. The relevant intrinsicparameters of such a pinhole camera are the focal length, the principalpoint and the optical distortion parameters. The focal length is thedistance between a center of projection (aka the pinhole) and a sensorplane. The principal point is the location on the digital sensor wherethe center of the idealized pinhole camera lies. The distortionparameters model the optical distortion imparted on the image by thenon-ideal optics of the stereoscopic visualization camera 12.Additionally, a tangential distortion parameter is modeled to attempt todescribe misalignment of the camera optics path with a line emanatingperpendicular to the plane of the image sensor. The extrinsic parametersobtained from the calibration include a translation and rotation, ortransformation of the target relative to the stereoscopic visualizationcamera 12 for each calibration image captured.

Referring to FIG. 2, camera calibration may be performed at a singlezoom (magnification), working distance W (and/or focal distance) settingof the stereoscopic visualization camera 12 using a calibration function57 executable by the controller C and/or the camera processor 52. In oneexample, the calibration function 57 may incorporate an open sourcecomputer vision (OpenCV) function for each sensor optical path of thestereoscopic visualization camera 12. A target used during suchcalibration must have discernible features of known relation to eachother and such features must be indexable in images of said target. Thatis, the location of each feature as found in an image of the target mustbe relatable to the actual known physical parameters of the target. Anexample of a calibration target 520 is shown in FIG. 8B. Whilecalibration target 520 is shown with a checkerboard pattern, it isunderstood that other patterns may be employed.

Images of the calibration target 520 are taken from various viewpointswith one of the 2D visualization modules V1, V2. In some embodiments thestereoscopic visualization camera 12 is moved in three dimensions andits orientation changed relative to the calibration target 520 using therobotic arm 24 of FIG. 1. This facilitates automation of the calibrationtask. In other embodiments, the calibration target 520 may be moved inthree dimensions and its orientation changed relative to thestereoscopic visualization camera 12 by hand or by using motorizedactuators. The multiple image captures from various viewpoints of thecalibration target 520 are sent into the calibration function 57 alongwith knowledge of its physical parameters such as width 504, height 506and respective gaps 508 and the number of internal intersections of e.g.dark and light squares, as well as the known parameters of the 2Dvisualization module such as pixel dimensions. This calibration resultsin a focal length and principal point location for that sensor asdescribed for the camera model. Such parameters are input into the VRmodule 51 to create two-dimensional images of the OCT data ready foraccurate blending or “fusing” with the data from the 2D visualizationmodules V1, V2 of the stereoscopic visualization camera 12. Thiscalibration also results in an estimate of optical distortionparameters.

Locating and Orienting the 2D Visualization Module Space Relative to theTarget Space

The space of the 2D visualization module (also called “camera space”) isalso referred to as the “center of projection” space or COP space 830.Referring to FIG. 11A, the location (or “position” or “origin”) andorientation of the target space 832 relative to the COP space 830 aretaken together and referred to herein as the second transformation 834of the target space 832 relative to the COP space 830. Besides the otherparameters described, the calibration function 57 is configured toobtain an estimate of the third transformation 862 of the calibrationtarget 520 relative to the COP space 830 for each image capture taken.Mathematically this information may be encoded in a transformationmatrix.

Referring to FIG. 11A, when the calibration target 520 is orientedorthogonal to the Z-axis of the COP space 830 and the stereoscopicvisualization camera 12 is focused on the calibration target 520, theplane of the calibration target 520 is a distance equal to a focaldistance F away from the center of projection 802 in the negative Zdirection. The origin of the calibration target 520 is shown to betranslated by a first distance 840 and 842 in the X and Y directions,respectively, relative to the camera space or COP space 830. Referringto FIGS. 11A and 12A, the part of the COP space 830 that is viewable bythe stereoscopic visualization camera 12 is bounded by rays 844, 846,848 and 850. The respective OCT extents 810 are shown in FIG. 12A. Thecalibration target 520 is located within the rays 844, 846, 848 and 850and nominally about the focal distance for each snapshot used in thecalibration.

The calibration function 57 of FIG. 2 may include a stereoscopicfunction to calibrate separately each of the two 2D visualizationmodules V1, V2 of the stereoscopic visualization camera 12. Thisdetermines the physical relationship between the two 2D visualizationmodules V1, V2, such as intraocular distance and relative COP locations,and enables subsequent rectification of image pairs from the two modulesinto a form in which it is straightforward to determine physicaldimensions in captured images when features can be found and matched inthe images from the two modules. Measurement of such dimensionsabsolutely from the images directly is thus enabled. Determination offeatures' absolute position and orientation relative to some referencepoint on the stereoscopic visualization camera 12 is also enabled andutilizes the disparity between the locations in the first and secondviews of the stereoscopic visualization camera 12 of a given object orfeature. Disparity is a function of how far away the object is from thestereoscopic visualization camera 12. This may be determined using aform of triangulation with the intraocular distance of the calibratedstereoscopic visualization camera 12.

By calculating the disparity of as many points in the stereoscopic imageas possible (as opposed to just a single feature), a disparity map isgenerated which is converted to a depth map which stores locationsrelative to the stereoscopic visualization camera 12 for all such imagepoints. With the OCT module calibrated and registered to thestereoscopic visualization camera 12, the image data is thus mapped tothe OCT space. This enables a variety of features such as obliqueviewing, which is described below with respect to FIGS. 13A, B and C.Such calibration thus provides several of the requirements ofcalibrating and registering the 2D visualization modules V1, V2 of thestereoscopic visualization camera 12 and the OCT module.

In one embodiment, performing camera calibration for each of the 2Dvisualization modules V1, V2 of the stereoscopic visualization camera 12includes the following: field of view calibration, optical axisdetermination relative to some frame of reference and focus distancedetermination along the optical axis from some center of projectionpoint to the focal plane, including absolute focus distance calibration.Field of view may be determined over zoom and working distance byimaging a target having known dimensions, e.g. a chessboard with knownindividual squares. Additionally, the corners of the squares are easilydetected by image processing algorithms. Further, calibrating theseparameters uses a means to alter the distance between the stereoscopicvisualization camera 12 and the target by controlled amounts. Furtherstill, for calibrating the optical axis and focus distance these amountsare measurable.

Alternative Method for Field of View Determination

The field of view at the focal plane of a real-world camera is one ofthe parameters needed to set up the stereoscopic volume renderer so thatits output will match visually the output of the stereoscopicvisualization camera 12. The field of view at the focal plane of each ofthe 2D visualization modules V1, V2 of the stereoscopic visualizationcamera 12 may be found in this alternative method as follows. Thestereoscopic visualization camera 12 of the head unit 18 is presumed tohave already been calibrated opto-mechanically such that the first andsecond optical paths are each in focus and aligned to convergevertically and horizontally at the focal plane for each zoom and workingdistance setting. As noted above, the calibration target 520 (see FIG.8B) may be placed on a table 780 (see FIG. 2), which can travelnominally along the optical axis of the stereoscopic visualizationcamera 12 (in the z-direction here). The position of the table 780 ismeasured and recorded at each such measurement. The calibration target520 may be positioned such that it nominally aligns with the cameraimage axes.

The stereoscopic visualization camera 12 may be set to a zoom andworking distance setting, with the settings being recorded. Thecalibration target 520 may be brought into focus either by moving thetable 780 or by changing the working distance setting of thestereoscopic visualization camera 12. A snapshot is taken of thecalibration target 520 and analyzed (e.g. by counting how many squaresare along the horizontal axis) to determine the horizontal field ofview, the field of view being defined as “the linear dimension (i.e.width) that the stereoscopic visualization camera 12 can see at thefocal plane for this zoom and working distance setting. The estimatedfield of view is stored in a table along with the current zoom andworking distance. To avoid distortion effects which are typically worseaway from the center of the image, in some embodiments only a centralportion of the image is used to make the estimate and the resultextrapolated to calculate the field of view dimension for the wholeimage. For example, if the center 25% of the image width is 10chessboard squares wide, then the whole image width (100%) can beextrapolated to be 40 chessboard squares wide. Note that the optics ofthe stereoscopic visualization camera 12 in this embodiment aremanufactured to tight tolerances which results in low amounts of suchdistortion. This means that omitting the distortion correction haslittle negative effect on the overall results obtained in using thisform of calibration. In some embodiments the center of the calibrationtarget 520 has different sized (e.g. smaller) component features suchthat when zoomed in (i.e. when at high magnification), the number offeatures is high enough to provide an accurate estimate of the field ofview. The process may be repeated at various zoom and working distancesettings across the range of both of those variables. During normal use,to get the field of view for any given zoom and working distance settinga lookup function interpolates among the closest (e.g. the closest four)values in the two-dimensional zoom/working distance space.

Method for Optical Axis and Focus Distance Calibration

The working distance W may be defined as a distance from the bottomsurface of the common objective lens set 72 along the optical axis (alsoreferred to as the view axis) to the scene or target site 16. Forexample, if a human viewer possessing generally pristine vision in botheyes looks at an object utilizing stereopsis, the apparent stereoscopicoptical axis may be considered as a line originating from, for example,the midpoint of a line from the center of one eye to the other. Oftenthe end of the nose is referenced for indication of the proximal end ofthe optical axis in scenes where the object is at a long distance.However, if the viewer closes one eye, the open eye's optical axisgoverns the view, and it is at a slightly different direction than thatof the stereoscopic optical axis. The two human views generally focusand converge at the object. In some embodiments, the two 2Dvisualization modules V1, V2 of the stereoscopic visualization camera 12may be set to focus and converge on the same point.

Focus distance may be defined as the distance from the focal plane ofthe stereoscopic visualization camera 12 to an origin or the center ofprojection along the optical axis of the stereoscopic visualizationcamera 12. The location and orientation of the optical axis may bedetermined during a calibration step in a known reference frame of thestereoscopic visualization camera 12, for example in the reference frameof the mounting stand 25 of FIG. 1. There are separate origins for eachmonoscopic view of the stereoscopic visualization camera 12. Thedistance from the focal plane of each of the 2D visualization modulesV1, V2 of the stereoscopic visualization camera 12 to the respectivecenter of projection for a pinhole camera model for each module is oneof the parameters needed to set up a virtual camera (such as in the VRmodule 51) such that its output will match visually the output of thereal-world camera. The origin is generally on the opposite side of thecommon objective(s) from the scene or target site 16 so focus distanceis usually greater than working distance by some offset, which offsetmight vary with working distance and or zoom. In the stereoscopicvisualization camera 12 working distance and thus focus distance changesmay be implemented by moving optical elements via motors and control.

The determination of the location and orientation of the optical axis insome known reference frame of the stereoscopic visualization camera 12as well as the position of the focal plane relative to the stereoscopiccenter of projection on that optical axis may be made as follows.Referring to FIG. 1, the stereoscopic visualization camera 12 andcalibration target 520 are mounted to the end of the robotic arm 24 viaa mounting stand 25. The link dimensions and joint configurations (e.g.sizes and angles) of the robotic arm 24 and mounting stand 25 are knownby design and/or by metrology at any given time. Thus, the position andorientation of the calibration target 520 relative to the stereoscopicvisualization camera 12 may be obtained relative to the mounting stand25.

In other embodiments, the stereoscopic visualization camera 12 may bemounted to a fixed base (e.g. coupling plate 26 of FIG. 1) and thecalibration target 520 mounted to a table (780 shown in FIG. 2)connected to a linear actuator 782. The position and orientation (andthus direction of travel) of the linear actuator 782 may be obtainedaccurately relative to the fixed base of the stereoscopic visualizationcamera 12. The direction of travel of the table 780 may be set by designto move nominally along the stereoscopic optical axis (the axialdirection A here) and in view of the stereoscopic visualization camera12 such that the calibration target 520 can be placed at will along thewhole working distance with the stereoscopic visualization camera 12viewing the calibration target 520 and the position read via thecontrols of the linear actuator 782. For highest precision, thestereoscopic visualization camera 12 may be set to maximummagnification.

Referring to FIG. 1, starting at a minimum value of the working distanceW (and thus the minimum focus distance), the robotic arm 24 isconfigured to move the stereoscopic visualization camera 12 to align thecenter of the calibration target 520 with the center of both the firstand second images (e.g. left and right) of the stereoscopicvisualization camera 12, to within some tolerance. This ensures thecalibration target 520 is centered and at the focal plane of thestereoscopic visualization camera 12. Precise orthogonality of thetarget site 16 and the optical axis is not required for this step. Thereal position and orientation of the stereoscopic visualization camera12 relative to the center point of the calibration target 520 may beobtained from the position of the robotic arm 24 and the dimensions ofthe plate and are recorded. The working distance W is then changed by aportion of the full working distance range by physically moving thestereoscopic visualization camera 12 with the robotic arm 24, and thenthe image is re-focused, e.g. by moving a focus motor. The process oftarget centering and position and orientation recording is repeated,with the resultant robot move recorded. The move changes are calculatedbetween sample points. The relative focus distances are thus determinedbetween the points; the absolute distance from the focal plane to thecenter of projection at any point relies on an “absolute focus distancecalibration” as described above. This is followed by another change inthe working distance W and subsequent re-centering, re-focusing andreading of the robotic arm position until the entire focal length rangeis spanned thereby sampling this range. Since a single point on thecalibration target 520 is used (the center here), the relativeorientation of the calibration target 520 and stereoscopic visualizationcamera 12 is not needed.

An orthogonal distance regression may be performed on this series of 3Dpoints to find the stereo optical axis with respect to some known frameon the stereoscopic visualization camera 12 for example the mountingmechanism reference frame. Projecting each of these points to this axisenables the working distance W to be calculated at each position of thefocus motor that was used in the focal length range sampling. Combiningthese sets of information along with the absolute focus distancecalibration and generating an interpolation algorithm, the position ofthe center of the image at the focal plane relative to the known frameon the stereoscopic visualization camera 12 is found across the entireworking distance range. For endpoints of the sample range, anextrapolation algorithm is generated to reach the extents of the rangenot reached by the sample points.

The stereoscopic visualization camera 12 is assumed to be par-centralover focus and zoom, and par-focal over zoom. If these assumptions donot hold, this procedure may have to be repeated over multiple zoomlevels and interpolated. If the stereoscopic visualization camera 12 isnot par-central over focus, a more complex model than the linear one maybe required. These assumptions do work for the accuracy requirements onthe stereoscopic visualization camera 12 currently in production.

A final step determines the “up” and “right” vectors of the stereoscopicvisualization camera 12 relative to the head unit 18. In one embodiment,crosshairs may be overlaid on the visualization display, such crosshairsbeing aligned with a row and a column of the data such that theyprecisely determine the “up” and “right” vectors of the camera images. Adigital overlay does this with pixel accuracy. The robotic arm 24 movesthe stereoscopic visualization camera 12 such that the center point ofthe calibration target 520 (see FIG. 8B) is followed along the verticalcrosshair and kept in focus. The movement of the robotic arm 24 isrecorded such that the “up” direction of the stereoscopic visualizationcamera 12 is now correlated with the known robot motion. The “right”direction is determined in a similar manner.

Absolute Focus Distance Calibration

The absolute distance from the center of the focal plane to the centerof projection may be estimated by placing the calibration target 520 infocus and noting the location on the calibration target 520 of the fourcorners of a rectangle of the display screen. Then using the table 780,the calibration target 520 is moved a known distance away from the focalplane, the camera optics are not changed (it is not refocused nor iszoom changed at all,) and the new locations on the chessboard of thesame four corners of a rectangle of the display screen are noted. Usingthe trigonometry method of similar triangles as well as additionalpositions of the table 780 and images, the position of the center ofprojection of each 2D visualization module relative to the calibrationtarget 520 is estimated.

The location and orientation of the calibration target 520 relative tothe table 780 and of the table 780 relative to the mount of thestereoscopic visualization camera 12 are known by design and bymetrology. Thus, the position of the center of projection (COP) of each2D visualization module relative to the reference frame of thestereoscopic visualization camera 12 is calculated. The choice oforientation of the “COP space” for each 2D visualization module dependson the application. In one embodiment it is chosen such that the originis at the center of the screen with the positive X-axis running to theright and the positive Y-axis running upwards onscreen, and the 2Dvisualization module is looking down the negative Z-axis (so thepositive Z-axis is coming out of the screen.) The whole system can beharmonized with simple mathematical adjustments via matrixmultiplication(s). In another embodiment, absolute focus distancecalibration is achieved by imaging a virtual shape, such as a wireframecube, by the 2D visualization modules V1, V2 of the stereoscopicvisualization camera 12 as well as the OCT module and stereoscopicallyrendering the output from each blended onscreen and manually alteringthe location of the center of projection of the volume renderer of theOCT data until the perspective of the cube matches for the two imagingmodalities.

V. System Registration

Registration may be understood as the alignment in rotation, translationand scale of the two views of the stereoscopic visualization camera 12to the respective views of the VR module 51 (embodying a virtual camera)rendering the OCT data, as well as matching the respective perspectives.While a few registration techniques are described below, it isunderstood that other techniques and methods may be employed.Registering the first set of volumetric data with the second set ofvolumetric data may include aligning a local area of interest in thefirst set of volumetric data in position, orientation and size with thesecond set of volumetric data. For example, referring to FIG. 5, thelocal area of interest may include a corneal limbus 332 and a scleralvasculature 334.

Registration may include finding the location and orientation of therespective centers of projection (COP) of the 2D visualization modulesV1, V2 of the stereoscopic visualization camera 12 relative to thelocation and orientation of the OCT data space. For simplicity only oneview is discussed. Finding the location and orientation of an objectrelative to another object is also known as finding the transformationof the coordinate system of the first object relative to the second, orvice-versa. Additionally, the coordinate system of the object is alsoknown as the “space” of the object. A third object, which remains fixedin relative location and orientation to both objects, may be used todetermine such a transform by using the inverse transform of one of thetransformations. An illustration of inverse transforms is below. Forexample, referring to compass 870 in FIG. 12B, point Q is 4 blocks eastof point P and point R is 2 blocks south of point P, and point R is 2blocks south and 4 blocks west of point Q. The directions to get frompoint P to point R would be to “go 2 blocks south” and the directions toget from point Q to point R would be to “go 2 blocks south and 4 blockswest.” The inverse of the directions to get from point Q to point Rwould be for example “go 4 blocks east and 2 blocks north” (in thisexample the inverse could also be “go 2 blocks north and 4 blocks east,”the result is the same). Thus, directions from points P to R followed bythe inverse directions from points Q to R would be “go 2 blocks souththen go 4 blocks east and 2 blocks north” which gets us from points P toQ.

In this embodiment, the third object used for this maneuver is thecalibration target 520 (see FIG. 8B) used elsewhere to calibrate the OCTmodule 14 and 2D visualization modules V1, V2. It is understood thatother objects or types of devices may be used. The common physical spacerequirement is satisfied by the calibration steps for each of the 2Dvisualization modules V1, V2 and the OCT module 14 which convert theiroutput data to real-world measurements e.g. in millimeters. The secondtransformation 834 (see FIG. 11A) of the calibration target 520 relativeto the 2D visualization module space (the origin of which is located atthe COP) may be referred to as “COP_T_TARGET” which is read “backwards”as “the transformation from target space to the center of projectionspace.” The first transformation 774 (see FIG. 10A) of the calibrationtarget 520 relative to the OCT space may be referred to as“OCT_T_TARGET” which is similarly read backwards as “the transformationfrom target space to OCT space.”

To enable registration of the OCT module 14 to each of the 2Dvisualization modules V1, V2 of the stereoscopic visualization camera12, the third transformation 862 (see FIG. 12A) is needed of the OCTspace to the 2D visualization module space, or COP_T_OCT (or itsinverse: OCT_T_COP.) COP_T_OCT may be calculated using the secondtransformation 834, first transformation 774 and the inverse maneuverjust described: COP_T_OCT=COP_T_TARGET*(OCT_T_TARGET). inverse

Here transformations are implemented mathematically in the controller Cand/or the camera processor 52 and/or the OCT processor 50, e.g. withcomputer graphics using matrices, and multiple transformations areimplemented by sequential multiplication of such matrices and theirinverses. Thus COP_T_OCT etc. are matrices in the actual computer codeimplementing these features. This equation implements registration ofthe two devices which when combined with proper calibration of theoptical parameters of the 2D visualization module and theOCT-data-to-real-world-measurements enables accurate fusion of data fromthe two spaces.

Perspective Calibration

Calibration of the 2D visualization modules V1, V2 of the stereoscopicvisualization camera 12 includes perspective determination whichdescribes how objects are sized relatively in the image when the objectsvary in distance to the 2D visualization modules V1, V2. This is becausethe three dimensions of the real world are projected onto atwo-dimensional plane in a 2D visualization module. This is called“perspective projection.” Objects of identical size appear in aprojected image at different sizes based on how far they are from theimager's effective center of projection. Perspective calibration is thematching of this relative size scale between the stereoscopicvisualization camera 12 and the OCT module 14 and is one of the criticalfactors in enabling accurate and realistic “fusion” of three-dimensionalimages from different visualization modules into a common imaging space.The OCT module 14 does not suffer from this “perspective projection”behavior because it is imaging the three-dimensional real world withthree dimensions; the image data resulting from the OCT imaging processis three-dimensional and is not projected onto two dimensions.

However, representing this three-dimensional data on a two-dimensionaldisplay, such as a flat panel display, requires that it be renderedusing a virtual camera such as a volume render camera characterized bythe VR module 51. The perspective projection of the VR module 51(virtual camera) needs to be set up to match that of its respective mateof the 2D visualization modules V1, V2 of the stereoscopic visualizationcamera 12. Note that a “3D” display is really just a 2D display whichallows different 2D images (“views”) to be presented to the user andtherefore representing three-dimensional data requires perspectiveprojection for each view.

The intrinsic and extrinsic parameters obtained during the cameracalibration of the 2D visualization modules V1, V2 may be used to set upeach respective volume render camera in the VR module 51 with a computergraphics language such that an item of known position and dimensions inthe real world visualized by the calibrated stereoscopic visualizationcamera 12 when drawn virtually in the computer graphics pipeline portionof camera processor 52, and when positioned at an identical position andorientation relative to the calibrated stereoscopic visualization camera12 will “match” the view of the physical object in size, shape andperspective. In a computer graphics renderer, modeling these parametersmay be implemented by setting mathematical properties such as aprojection matrix and a model-view matrix for controlling how a givenstructure is rendered onscreen including size, position, orientation andperspective.

VI. Oblique Visualization

Referring to FIG. 2, the system 10 may include an oblique visualizationmodule 55 selectively executable by the controller C to produce anoff-axis view of the shared composite view such that a respectiverepresentation of the first set of volumetric data and the second set ofvolumetric data is viewed at a predefined oblique angle. FIGS. 19A-13Crespectively illustrate a shared composite view of an eye E on a display32 (overlaid on a live stereoscopic image L). FIG. 13A shows a top downview 1300 of the eye E with a zero-degree oblique angle. FIG. 13Aillustrates an off-axis view 1330 of the eye E, in accordance with afirst oblique angle 1332. FIG. 13C illustrates an off-axis view 1350 ofthe eye E, in accordance with a second oblique angle 1352. In theexample shown, the first oblique angle is about 45 degrees and thesecond oblique angle is about 80 degrees.

As described above, the OCT module 14 gathers three-dimensional datafrom a volume in the target site 16. Referring to FIGS. 13A-C, the OCTmodule 14 images the nearly transparent features of the eye E, such asthe cornea 1302, lens 1304 and lens capsule 1306. Execution of theoblique visualization module 55 causes the controller C to: form avirtual image, such as the virtual image 804 in FIG. 11A, based in parton the first set of volumetric data from the OCT module 14. For example,a subset of scan positions may be employed to construct the virtualimage. The virtual image may be rendered, via the VR module 51, as afirst pair of stereo images.

Referring to FIGS. 13A-C, the visible, generally opaque, features of theeye such as the sclera 1308 and iris 1310 are captured by thestereoscopic visualization camera 12 and disparity may be used to formthe second set of volumetric data. The second set of volumetric data isrendered as a second pair of stereo images, via the VR module 51. Thefirst pair of stereo images and the second pair of stereo images areoverlaid or fused to form the off-axis view 1330 (or off-axis view1350). Since the view of the stereoscopic visualization camera 12 isstereoscopic, calibrated and registered to the OCT view, stereoscopicvisualization camera 12 is moved correctly as if it is viewed from thesame arbitrary angle. The predefined oblique angle may be selectable bya user via the user interface 54.

The system 10 enables viewing and user sensing of the “depth” dimensionof the eye E, which in turn, enables better control of instruments alongthe depth. For example, referring to FIG. 13C, the depth of a surgicaltool 1312 may be most accurately imaged in the off-axis view 1350,relative to the central axis view 1300. The surgical tool 1312 may havea tip portion 1314 configured for insertion into the eye E via anincision 1316. The depth sensing via the selectable oblique angle makesit easier to keep the surgical tool 1312 from contacting or puncturingtissue, such as the anterior chamber.

In order to visualize nearly transparent features in the target site 16(not visible to the stereoscopic visualization camera 12), portions ofthe OCT data may be volume-rendered as a stereo pair with nominallyidentical virtual camera parameters as the visible data. The renderingby the VR module 51 may be updated with every new frame of data. Forexample, the stereoscopic visualization camera 12 view may data updatesat a frequency of nearly 60 Hz. The OCT data may update between 3 Hz and30 Hz. These updates are independent and may or may not be synchronized.The rendering may also update to a new oblique angle when a new cameraposition and orientation is input by the user or other input. The outputimages of the stereoscopic visualization camera 12 may continue to berendered in the same place relative to the OCT data.

VII. Surgical Applications

The system 10 may be employed in various surgical applications for ahost of purposes: to enhance visualization, to evaluate the opticalcharacteristics of the eye before, during and after surgical changes, toassist specification and placement of intraocular devices, to easecomplications and to enable overall better patient outcomes. It is to beunderstood that the system 10 may be configured to collect datapre-operatively, intra-operatively, or peri-operatively. A number ofspecific applications are described below.

Extracting Structural Points and Surfaces of the Cornea

Referring to FIG. 14A, a schematic fragmentary perspective view of aneye E having a pupil 1402 and iris 1404 is shown. A plurality of depthscans 1406 may be employed to extract structural features of the eye E,extending across a first scan range 1408 and a second scan range 1410.Referring to FIG. 14B, a sectional view of the eye E is shown, alongwith output images 1420, 1430, 1460 and 1440, obtained from theplurality of depth scans 1406.

The system 10 may be employed to extract the position of the firstcorneal surface 1412 and the second corneal surface 1414. Using thesystem 10 of FIGS. 1 and 2 to sample a first row scan (B-scan) along thescan range 1408, for example, the sample starting points 1422, 1424,1426 appear in the output image 1420 as respective image points 1432,1434, 1436. The respective image points 1432, 1434 and 1436 representsample points of the first corneal surface 1412.

Referring to the output image 1430, the appearance of a bright or highintensity pixel, such as respective image points 1432, at a distance1438 from the top edge of the output image 1430 may be converted to athree-dimensional location corresponding to the location of the samplestarting point 1422 in the reference frame of the eye E. Each of theplurality of depth scans 1406 define respective starting points. Thethree-dimensional locations corresponding to the respective startingpoints of the plurality of depth scans 1406 may be similarly found,collected as a point cloud and converted to a single surface via a curvefitting technique. Referring to output image 1440 in FIG. 14B, thesample points 1444 may be fitted to a curve 1442.

As described previously, when the OCT module 14 and stereoscopicvisualization camera 12 are registered, the three-dimensional positionof a voxel in the respective data space of the OCT module 14 may beobtained. This allows the three-dimensional real-world position of eachvoxel to be obtained relative to a reference point on the stereoscopicvisualization camera. A pixel's three-dimensional position is availableas long as its three-dimensional location in the OCT data space isknown, as it is by knowing, for example its X and Y locations in thetwo-dimensional image as well as the index of the transverse slice thatthe image represents. Hence the location of each pixel in the outputimage 1420 may be converted to a real-world position in the referenceframe of the stereoscopic visualization camera 12 and overlaid fordisplay in the shared composite view.

When the first beam B1 of the OCT module 14 passes through a material toresult in a detected surface point, the index of refraction of thematerial must be accounted for in that part of the light path duringcalculation of such distances. For example, light travels slower throughplate glass than through the human cornea, so for two such structures ofidentical thickness, the resultant OCT data points reflecting off thefront and rear surfaces of two such structures will be closer togetherfor the cornea than for the plate glass. Accordingly, a refractive modelgenerator available to those skilled in the art may be employed togenerate a full or partial optical model of the patient's eye duringsurgery to enable assessment and subsequent improvement of opticalaberrations present in the eye. The refractive model generator mayemploy the first set of volumetric data from the OCT module 14 such asthe first volumetric sample 1452, second volumetric sample 1454 andthird volumetric sample 1456 of the first corneal surface 1412, shown inFIG. 14B. The first volumetric sample 1452, second volumetric sample1454 and third volumetric sample 1456 appear respectively in the outputimage 1460 as image points 1462, 1464, 1466. The refractive modelgenerator may further employ the fourth volumetric sample 1472, fifthvolumetric sample 1474 and sixth volumetric sample 1476 of the secondcorneal surface 1414, shown in FIG. 14B, which appear respectively inthe output image 1460 as image points 1482, 1484, 1486. Additionally,the axial length 1560 of the eye E (see FIG. 15) may be employed.

Retinal Surgery: Visualization of Surface and Subsurface Features of theRetina

FIG. 15 is a schematic fragmentary sectional view showing the use of thesystem 10 to extract a 3D volumetric representation of an eye E having alens 1416. At the portion 1530 of the retina, shown enlarged in FIG. 15,the higher-resolution, lower depth OCT configuration is used, with thenominal OCT working distance residing in or near the retina. Referringto FIG. 15, a plurality of depth scans 1502, 1504, 1506, 1508 or rowscan 1532, are taken along the eye E. Referring to FIG. 15, the samplingpoint 1534 measures the front retinal surface 1510, while secondsampling point 1536 images the rear retinal surface 1512. Thesub-surface sampling points 1538, 1540 1542, 1544 image layers of theretina which here are showing a pathology 1550. The representation ofthe pathology 1550 is overlaid on the live view L such that the drawnpathology 1550 lines up with the pathology location in the live view L.This requires registration of the OCT module 14 and the stereoscopicvisualization camera 12 as described above. The surgical team may locatepathologies and other objectives while viewing the eye E in the sharedcomposite view, all on the same display. The surgical tools used arealso visualized in the live image L. The pathology 1550 is not visiblevia a traditional microscope. The system 10 enables visualization of thepathology 1550 and hence more effective treatment. The integration,calibration and registration of the OCT module 14 and the stereoscopicvisualization camera 12 enable greatly simplified location and guidancefor the user.

Controlling Image Resolution and Range

Referring to FIG. 2, a user may select between a larger sample depthwith a lower depth resolution and a smaller sample depth with higherdepth resolution, via tradeoffs among the spectral width of the firstlight source 40 at the spot scan 60, the optical characteristics of thespot scan 60 and the resolution of the detector 44. For retinalsurfaces, a depth scan with large sample depth with lower depthresolution may be employed. Referring to FIG. 15, such a depth scanpasses through the pupil of the eye E, but also could potentially passthrough the lens 1416, thereby sampling first and second lens points1518 and 1520 on the lens 1416. Additionally, the vitreous humor 1526may have some optical properties that are less than fully transparent.In some cases, due to surgical removal, the lens 1416 will not bepresent, similarly with the vitreous humor 1526, such that thedetermination of surface reflections is simplified somewhat. The spotscan 60 is movable about the target site 16 (see FIG. 2) using thesteering unit 64 to within the limits of the OCT module 14.Additionally, the OCT range and resolution (e.g. low-depth/highresolution, etc.) are modifiable by switching among different opticalsetups in the OCT module 14 itself. This switching is fast, on the orderof tens of milliseconds. With the integrated stereoscopic visualizationcamera 12 and OCT module 14 mounted on the robotic arm 24, the OCTsample volume may be moved about the target site 16 in a controlledfashion. This extends the OCT range significantly in the axial directionA and in the first and second transverse directions T1 and T2. The OCTreadout and robot position may be recorded in such a way, for example,by including a universal timestamp of sufficient accuracy such that theymay be later synchronized. From the registration process, the positionand orientation of the target site 16 relative to a known reference onthe stereoscopic visualization camera 12 are available, and the positionand orientation of the reference are in turn known relative to the robotreference frame, the position and orientation of the target site 16relative to the robotic arm may be obtained and controlled during anophthalmic procedure employing such a range extension.

Measuring Axial Length of the Eye

Referring to FIG. 15, the system 10 may be employed to determine anaxial length 1560 of the eye E, which is the distance from the firstcorneal surface 1412 to the front retinal surface 1510. Referring toFIG. 15, the axial length 1560 may be extracted by the difference inreal-world coordinates between the measurement point 1514 on the firstcorneal surface 1412 and the measurement point 1522 on the front retinalsurface 1510.

Axial length measurement of the patient's eye changes as a function ofwhere the patient's eye is focused. In order to have a consistentmeasurement, the patient may be asked to focus on a fixed landmark builtinto the system 10. Alternatively or in addition, the axial lengthmeasurement may be taken continually during the surgery, for example byswitching the OCT module 14 back and forth between its “low-depth,higher resolution” mode used to measure the first corneal surface 1412and the second corneal surface 1414, and a “high-depth, lowerresolution” which allows for measurement of the axial length 1560. Byadjusting the steering unit 64 of FIG. 2, the spot scan 60 may be movedsuch that multiple measurements may be made to obtain an average value.

Extracting Curvature of the Cornea

Referring to FIG. 16B, the curvature of the front surface (or rearsurface) of the cornea 1602 of the eye E may be obtained with the use ofone or more row scans 1604 or slices covering the surface of the cornea1602, such as first slice 1610, second slice 1612, third slice 1614,fourth slice 1616 and fifth slice 1618. FIG. 16A shows the output imageof the first slice 1610. The cornea 1602 is transparent orsemi-transparent at the wavelengths used in the first beam Bland has adimension that is less than the full depth range (see detected depth 204in FIG. 4A) of the first beam B1. Thus a depth scan may be configured toencounter the cornea 1602 multiple times, with the sample path resultingin the detection of at least two reflection points along the path. Forexample, the first, second and third starting points 1622, 1624 and 1626(of FIG. 16B) appear respectively in the output image 1620 of FIG. 16Aas first, second and third image points 1632, 1634 and 1636.

Referring to FIG. 16B, the row scans 1604 or slices may be extended orcontinued along a scan range 1606 to obtain or map further surfacepoints. FIG. 16C shows a superimposed image 1640 of the respectiveoutputs obtained from the first slice 1610, second slice 1612, thirdslice 1614, fourth slice 1616 and fifth slice 1618. To reduce noise inthe image, noise reduction techniques may be employed. For example,thresholding on the intensity data may be performed by discardingoutlying values below a low threshold and/or above a high thresholdvalue. The outlying values may be set empirically using actual datasets.

FIG. 16D is a schematic diagram showing a modeled or extracted surface1650, obtained from the row scans 1604 of FIG. 16B. The controller C ofFIG. 2 may be configured to first locate the reflection points in thefirst slice 1610, such as first, second and third starting points 1622,1624 and 1626 in FIG. 16B. The reflection points in the (last) fifthslice 1618, such as fourth, fifth and sixth starting points 1642, 1644and 1646 (see FIGS. 16B and 16D) are also obtained, as well as therespective starting points in the intermediate slices. For simplicity,only one surface of the cornea is shown in FIG. 16D. The collection ofstarting points may be fitted to a surface which matches priorassumptions. With enough detected points, a surface model 1652 is fittedto the points. The type or functional nature of the surface model 1652may be varied. For simple curvature along a single B-scan direction ofOCT data, a single two-dimensional slice of OCT is sufficient. Toestablish the maximum and minimum such curvatures, for example for thehuman cornea, multiple row scans 1604 may be used along the radialdirection of the eye E. In such a manner, the curvature of both thefirst corneal surface 1412 and the second corneal surface 1414 (see FIG.14B) along the scan range 1606 may be obtained, enabling estimation ofthe optical characteristics of the detected surfaces.

The curvature of the first corneal surface 1412 and the second cornealsurface 1414 may change during the surgery due to incisions, removal oftissue, pressure changes and other reasons. Some of the changes may beintentional to offset or correct some existing irregularities in theeye. The surface curvature estimates may be done multiple times persecond, providing the surgeon with up-to-date information as the surgeryproceeds and as changes are made to the surfaces. This enables betterpatient outcomes as final incision results may be evaluated andadjusted. Using the faster “two B-scan” process described above alsoenables faster updates of this measurement when the scans are positionedalong or nearly along the paths of such curvature on the eye andtracking is used to keep the scans so positioned.

By adjusting the steering unit 64 of FIG. 2, the spot scan 60 may bemoved such that relevant portions of the desired structure are measured,for example, where there is sufficient curvature information. However,in human eye surgical scenes, the patient is typically awake andtherefore the eye can move about the scene during the procedure. Theoptical center of the eye may found prior to the case, located withcompensation due to cyclotorsion at the start of the case, and indicatedwith annotation over the live image L, such as for example on the sharedcomposite view 300 of FIG. 5. While tracking various eye features in thelive image L, the second annotation 330 may be kept in place onscreenrelative to the eye E as the eye moves about naturally during thesurgery. In some embodiments, the spot scan 60 may be moved with thistracking, keeping it located largely sampling the same volume of the eyeE during the case. In other embodiments the OCT data itself is used totrack the position of the eye by comparing frames near in time to eachother and using a three-dimensional feature tracker to determine theamount of movement between frames. The “white-to-white” distance of thehuman eye is the horizontal diameter of the cornea. In some embodiments,the transverse range of the OCT module 14 may exceed this number, whichenables the OCT sample volume when properly positioned to contain thecornea completely in the two transverse dimensions. Additionally, therange of radius of curvature of the cornea in humans results in aneffective depth that is within the axial range of the OCT module 14.Thus, the OCT module 14 in some embodiments is able to sample therelevant regions of the full cornea including the whole front surfaceand rear surface.

The controller C may be configured to separately extract the first andsecond corneal surfaces 1412, 1414 (see FIG. 14) of the eye E capturedby the OCT module 14 and generate a respective topographic map generatedfor each surface. The respective topographic map may be overlaid withthe view from the 2D visualization modules V1, V2 of the stereoscopicvisualization camera 12. Referring to FIG. 5, a live (stereoscopic)image L of the eye E is overlaid with a topographic outline 312, andshown on the first display 32. The shared composite view 300 of FIG. 5may use different patterns to represent relative depth. The topographicoutline 312 may be represented by a plurality of topographic levels,such as a first topographic level 314 and a second topographic level316. The number of possible topographic levels is related to the axialresolution of the OCT module 14. The controller C may be configured toadd at least one annotation 320 (dashed circle in FIG. 5) showing aboundary of the two-dimensional OCT view over the shared composite view300. A second annotation 330 may indicate a landmark of the eye E. Thecontroller C may be configured to maintain the relative position of thesecond annotation 330 on the first display 32 relative to other featuresof the eye E.

Astigmatism Correction: Extraction of Steep and Flat Axes of anAstigmatic Cornea

Astigmatism in the human eye results partly from the cornea being shapedsomewhat similar to an ellipsoid, with two marked curvature linesinstead of a spherical shape with a uniform curvature across itssurface. The system 10 may be employed to ascertain the degree ofastigmatism in a patient, both as a diagnostic tool and during acorrective procedure in real time. In one example, referring to FIG.17A, a plurality of depth scans 1702 of the cornea 1704 of an eye E aretaken. The plurality of depth scans 1702 may be arranged in a starpattern 1706, with a plurality of row scans 1708 each crossing a center1710, shown in FIG. 17A. While in the example shown in FIG. 17A, thereare five row scans 1708, it is understood that any number may beemployed. Additionally, other types of scan patterns may be employed.

FIG. 17B shows the cornea 1704 with multiple spot scans 60. Referring toFIG. 17B, the curvatures of interest extracted from the multiple spotscans 60 are a steep meridian 1720 and a flat meridian 1730, which aredrawn as lines tracing the curvatures respectively, similar to lines oflongitude on a globe. The steep meridian 1720 and the flat meridian 1730may be found by tracking the maximum and minimum points of curvature onthe cornea 1704. By using a star pattern 1706 shown in FIG. 17A, alongwith positioning the spot scan 60 approximately near the apex of thecornea 1704, each of the plurality of row scans 1708 will roughlytransit these curvatures of interest, simplifying detection of the apex,the steep meridian 1720 and the flat meridian 1730. The steep meridian1720 and the flat meridian 1730 may be drawn with different brightness,color/shades and/or patterns, for example, to allow the surgical team todifferentiate from the background scene. Once an initial trace of thesteep meridian 1720 and the flat meridian 1730 are made, the OCT module14 may be placed into a faster “two B-scan” mode as described above, andthe steep meridian 1720 and the flat meridian 1730 may be measured at afaster rate, allowing more real-time feedback and adjustments. Thisenables estimation of the extent of astigmatism of the eye E, enablingbetter selection and alignment of corrective lenses. The measurementswith the system 10 may be made at time of surgery after some correctiveincisions have been made, allowing for production of a fully custom lenstailored to this specific patient by, for example, 3D printing orotherwise manufacturing a lens while the patient is undergoing surgery.

Cataract Procedure: Adjustment of Intraocular Lens in Real-Time

The system 10 may be employed for guiding an adjustment and subsequentimplantation of an intraocular lens in real-time. FIG. 18A is aschematic diagram of an example intraocular lens 1802 configured forimplantation into an eye, illustrated with a first lens axis 1804 and asecond lens axis 1806, and centered at a lens origin 1808. Theintraocular lens 1802 includes a first arm 1810 and a second arm 1812for positioning the intraocular lens 1802. Referring to FIG. 18B, afirst shared composite view 1820 of the eye E is shown on the firstdisplay 32 (or second display 34 of FIG. 1). FIG. 18B shows the pupil1822 and the iris 1824. A first corneal axis 1826 and a second cornealaxis 1828 and an eye center 1830 are drawn as overlays over the liveimage L, along with the intraocular lens 1802. In the first sharedcomposite view 1820 of FIG. 18B, the first lens axis 1804 and the secondlens axis 1806 are not sufficiently aligned with the first corneal axis1826 and the second corneal axis 1828.

For an astigmatic patient undergoing a cataract procedure, theintraocular lens 1802 may be especially chosen to correct theastigmatism. For example, toric lenses are designed to be complimentarythe patient's astigmatism, and must be aligned so that the complimentary(i.e. corrective) nature of the astigmatism correction is maximized. Forbest performance in both astigmatic as well as non-astigmatic patients,the intraocular lens 1802 is centered on the patient's visual axis sothat the intraocular lens 1802 corrects the patient's vision as well aspossible. To facilitate alignment of the first lens axis 1804 and thesecond lens axis 1806 with the first corneal axis 1826 and the secondcorneal axis 1828, surgeons may make physical markings on the eye todefine a known axis to guide them during rotational adjustment of theintraocular lens. However, these markings may smudge and produceinaccuracies.

FIG. 18C shows a second shared composite view 1840, using the curvatureof the live surfaces extracted via the system 10 (as described abovewith respect to FIGS. 16A-D) to guide the placement of the intraocularlens 1802. Referring to FIGS. 18B and 18C, an annotated curvature 1832may be drawn over or overlaid onto the live view L. The annotatedcurvature 1832 may be used by the surgeon to guide the movement of theintraocular lens 1802 and improve alignment.

The overlays may be updated with latency. As noted above, thestereoscopic visualization camera 12 and the OCT module 14 may define arespective latency. To mitigate this, the controller C may include afirst set of image buffers configured to selectively delay the displayof the two-dimensional OCT view in order to match the respective latencyof the stereoscopic visualization camera 12. The controller C mayinclude a second set of image buffers configured to do the opposite, andselectively delay the display of the two-dimensional stereoscopic viewto match the respective latency of the OCT module 14. This enables thesurgeon to align the first lens axis 1804 and the second lens axis 1806with the first corneal axis 1826 and the second corneal axis 1828,thereby optimizing the implant procedure and resulting in better patientoutcomes.

Alternatively, landmarks or features of the intraocular lens 1802 may bespecified manually by a user, “locked in place” and tracked by the OCTmodule 14 and the stereoscopic visualization camera 12 during theophthalmic procedure. The landmarks may include the lens origin 1808,and the respective location on the body of intraocular lens 1802 wherethe first arm 1810 and the second arm 1812 exit the body. The cornealshape may be changed during surgery, often intentionally. Theintraoperative characterization of the corneal shape enabled by thesystem 10 also enables optimum specification of the intraocular lens1802 to be implanted during a cataract procedure.

As noted above, intra-operative data obtained by the OCT module 14 maybe employed to build a three dimensional model of the refractivesurfaces during surgery. A refractive model generator algorithmavailable to those skilled in the art may be employed. By overlaying areconstruction of this refractive model on top of the live view L, thesurgeon may be able to assess the extent of astigmatism in real-time,thus helping to optimize the lens placement and orientation.Additionally, with the three dimensional model of the refractivesurfaces being generated at the time of surgery, the parameters of theintraocular lens 1802 for implantation may be calculated more accuratelythan by other methods.

Corneal Transplants/Surgery: Visualization of Defects

A common cause of a poor outcome in corneal transplant procedures is atorn, folded, or wrinkled donor cornea placed inside the patient's eye.Because the cornea is optically transparent or semi-transparent, it maybe extremely difficult to see these types of tears, folds or wrinkles. Asurgeon may be able to see a tear, fold, or wrinkle with an OCT image,however, they will not necessarily know or be able to pinpoint thelocation of the abnormality when viewing the eye through a microscope.

FIG. 19 illustrates visualization of a complication in a cornealprocedure. Referring to FIG. 19, a schematic diagram of a sharedcomposite view 1900 of an eye E, having a pupil 1902 and iris 1904, isshown. When the OCT module 14 and the 2D visualization modules V1, V2,of the stereoscopic visualization camera 12 are registered, the overlay(and fusing) of volume-rendered OCT data with the live view L onto thefirst display 32 (and/or second display 34), enables detection of andresponse to pathologies present in tissue that is difficult to visualizeusing traditional imaging modes.

During a corneal transplant the surgeon's goal is to remove by incisionthe patient's existing cornea (if present) and replace it with a donorcornea 1906 cut to approximately the same shape as the removed cornea.FIG. 19 illustrates the corneal transplant border 1908. As the cornea isa thin tissue, complications may arise during the transplant that aredifficult to visualize. For example, referring to FIG. 19, the tissuemay fold over itself, resulting in a pathological region 1910. Thesystem 10 (see FIGS. 1 and 2) may detect multiple surfaces and layerssimultaneously via a row scan 1912 (i.e. a B scan composed of aplurality of depth scans 202 as described above). The controller C maybe configured to isolate the pathological region 1910 as being between afirst border depth scan 1914 and a second border depth scan 1916, e.g.via the detection of multiple edges. The pathological region 1910 hasmultiple edges present in the fold which results in more “hits” for thefirst beam B1 (see FIG. 1) and a region of high pixel intensity. Whilethe pathological region 1910 shown in FIG. 19 is a folded region, otherpathologies such as tears, rips and wrinkles may be similarly isolatedand visualized, enabling immediate detection and correction of suchproblems. For example, wrinkles have multiple surfaces while tears havediscontinuities.

Referring to first output image 1920 from the row scan 1912 in FIG. 19,a top surface 1922 of the eye E may be imaged along with first portion1924, second portion 1926 and third portion 1928. Referring to thesecond output image 1930 in FIG. 19, a curve 1932 may be fit along suchsurfaces. In this example, the controller C may be configured to displayor annotate or draw in an overlay (solid line in FIG. 19) at thelocation of the pathological region 1910 for improved visualization.Additionally, the volume render process along with the registrationmethod described above may enable drawing of a useful overlay withoutcurve fitting. As more edges are present in a fold, this results in more“hits” for the first beam B1 and thus more “hits” for the cast virtualrays in the VR module 51. This enables a higher resultant value alongthose virtual rays for the pathological region 1910 compared to theirsurroundings, enabling the VR module 51 to differentiate visually theseareas. The controller C may be configured to depict or visualize thepathological region 1910 with onscreen pixels of different brightness orcolor compared to the surroundings.

By employing a shared composite view as described above, and furtherhighlighting or annotating the area where the pathological region 1910is found, the surgeon may be shown precisely where they need to directtheir attention. When combined with real-time OCT, this will furtherallow them to precisely manipulate the donor tissue to smooth out anyfolds or wrinkles, and to close together (as well as possible) any tearsin the tissue. This method also works to enable or greatly improve thevisibility of other difficult- or impossible-to-visualize tissue such assubsurface retinal lesions, tears, detachments and other pathologies.

Other Issues in Visualizing Eye Characteristics in Real-Time DuringSurgery

As noted above, latency overlays may be drawn over or intertwinedappropriately with the live image L at a rate sufficient for theoverlays to stay aligned in position and orientation to within anacceptable tolerance on moving objects to which they are meant to bealigned. There is a latency between the movement of the eye E and theupdate of the position and/or orientation and/or scale and/orperspective of the overlay meant for the eye E. This latency results inslight misalignment in such parameters, but reduces to the minimumtolerance for each parameter when the eye E stops moving (that is theoverlay “catches up” to the object once the object stops moving). Thelatency is related to system speeds, such as imaging frame rates,display pipeline processing time etc. In some embodiments a set of imagebuffers are included in both imaging modality pipelines such that thelatency of one modality relative to the other can be reduced oreliminated by delaying display of the other modality.

The system 10 enables tracking of various features of the eye E such asthe viewing vector and the apex of the cornea. Such features are used todetermine the “red reflex vector” which is the vector along which thesecond beam B2 from the stereoscopic visualization camera 12 isreflected from the patient's retina back into the stereoscopicvisualization camera 12. Red reflex may be employed by surgeons toilluminate the patient's eye structures from within, for betterdiagnostics and improved procedure facilitation.

The controller C of FIG. 1 may be an integral portion of, or a separatemodule operatively connected to, other controllers integrated with theOCT module 14 and stereoscopic visualization camera 12. The controller Cincludes a computer-readable medium (also referred to as aprocessor-readable medium), including a non-transitory (e.g., tangible)medium that participates in providing data (e.g., instructions) that maybe read by a computer (e.g., by a processor of a computer). Such amedium may take many forms, including, but not limited to, non-volatilemedia and volatile media. Non-volatile media may include, for example,optical or magnetic disks and other persistent memory. Volatile mediamay include, for example, dynamic random-access memory (DRAM), which mayconstitute a main memory. Such instructions may be transmitted by one ormore transmission media, including coaxial cables, copper wire and fiberoptics, including the wires that comprise a system bus coupled to aprocessor of a computer. Some forms of computer-readable media include,for example, a floppy disk, a flexible disk, hard disk, magnetic tape,other magnetic media, a CD-ROM, DVD, other optical media, punch cards,paper tape, other physical media with patterns of holes, a RAM, a PROM,an EPROM, a FLASH-EEPROM, other memory chips or cartridges, or othermedia from which a computer can read.

Look-up tables, databases, data repositories or other data storesdescribed herein may include various kinds of mechanisms for storing,accessing, and retrieving various kinds of data, including ahierarchical database, a set of files in a file system, an applicationdatabase in a proprietary format, a relational database managementsystem (RDBMS), etc. Each such data store may be included within acomputing device employing a computer operating system such as one ofthose mentioned above and may be accessed via a network in one or moreof a variety of manners. A file system may be accessible from a computeroperating system and may include files stored in various formats. AnRDBMS may employ the Structured Query Language (SQL) in addition to alanguage for creating, storing, editing, and executing storedprocedures, such as the PL/SQL language mentioned above.

The detailed description and the drawings or FIGS. are supportive anddescriptive of the disclosure, but the scope of the disclosure isdefined solely by the claims. While some of the best modes and otherembodiments for carrying out the claimed disclosure have been describedin detail, various alternative designs and embodiments exist forpracticing the disclosure defined in the appended claims. Furthermore,the embodiments shown in the drawings or the characteristics of variousembodiments mentioned in the present description are not necessarily tobe understood as embodiments independent of each other. Rather, it ispossible that each of the characteristics described in one of theexamples of an embodiment can be combined with one or a plurality ofother desired characteristics from other embodiments, resulting in otherembodiments not described in words or by reference to the drawings.Accordingly, such other embodiments fall within the framework of thescope of the appended claims.

What is claimed is:
 1. A system for guiding an ophthalmic procedure, thesystem comprising: a housing assembly having a head unit configured tobe at least partially directed towards a target site in an eye; anoptical coherence tomography (OCT) module at least partially located inthe head unit and configured to obtain a first set of volumetric data ofthe target site; a stereoscopic visualization camera at least partiallylocated in the head unit and configured to obtain a second set ofvolumetric data of the target site, the second set of volumetric dataincluding first and second views of the target site; a controller incommunication with the stereoscopic visualization camera and the OCTmodule; a volumetric render module selectively executable by thecontroller; wherein the controller has a processor and tangible,non-transitory memory on which instructions are recorded, execution ofthe instructions causing the controller to: register the first set ofvolumetric data from the OCT module with the second set of volumetricdata from the stereoscopic visualization camera to create a third set ofregistered volumetric data; render the third set of registeredvolumetric data to a first region to obtain a two-dimensional OCT view,via the volumetric render module; render the second set of volumetricdata from the stereoscopic visualization camera to a second region toobtain a live two-dimensional stereoscopic view, via the volumetricrender module; and overlay the first region and the second region toobtain a shared composite view of the target site and at least one ofvisualize and extract features of the target site.
 2. The system ofclaim 1, wherein the controller is configured to: obtain a plurality ofdepth scans extending through a corneal surface, each of the pluralityof depth scans defining respective starting points; collect respectivethree-dimensional locations corresponding to the respective startingpoints of the plurality of depth scans as a point cloud; and convert thepoint cloud to obtain an extracted curvature, including interpolatingbetween the respective starting points.
 3. The system of claim 2,wherein: the extracted curvature is characterized by a plurality ofdepths; and the controller is configured to visualize the sharedcomposite view with a plurality of topographic levels; and the pluralityof topographic levels respectively represent the plurality of depthssuch that the extracted curvature may be visualized.
 4. The system ofclaim 2, wherein: the ophthalmic procedure is a cataract surgeryincluding implantation of an intraocular lens into an eye; thecontroller is configured to add at least one annotation over the sharedcomposite view on a display, the at least one annotation indicating theextracted curvature; to maintain a relative position of the at least oneannotation in the shared composite view; and employ the extractedcurvature to guide alignment of an intraocular device to the eye.
 5. Thesystem of claim 4, wherein: the first set of volumetric data isconfigured to be updated at a first frequency and the second set ofvolumetric data is configured to be updated at a second frequency; andthe updating of the first set of volumetric data and the second set ofvolumetric data is synchronized to facilitate the alignment of theintraocular device to the eye.
 6. The system of claim 1, wherein: thecontroller is configured to obtain respective axial length measurementsin real time repeatedly during the ophthalmic procedure by switching theOCT module between a first resolution mode and a second resolution mode.7. The system of claim 1, wherein: the ophthalmic procedure is a cornealtransplant and the controller is configured to obtain a plurality ofdepth scans of the cornea; the controller is configured to identify andisolate a pathological region as being between a first one of theplurality of depth scans and a second one of the plurality of depthscans; and the controller is configured to add at least one annotationover the shared composite view on a display, the at least one annotationindicating the pathological region.
 8. The system of claim 1, wherein:the ophthalmic procedure includes astigmatism correction; the controlleris configured to obtain a plurality of row scans of a cornea; and thecontroller is configured to extract a steep meridian and a flat meridianfrom the plurality of row scans, via tracking of respective maximum andrespective minimum points of curvature on the cornea.
 9. The system ofclaim 8, wherein: the plurality of row scans is arranged in a starpattern.
 10. The system of claim 8, further comprising: a robotic armoperatively connected to and configured to selectively move the headunit; and wherein the robotic arm is selectively operable to extend aviewing range of the OCT module in an axial direction, a firsttransverse direction and a second transverse direction.
 11. The systemof claim 8, wherein: the stereoscopic visualization camera and the OCTmodule define a respective latency; the controller includes a first setof image buffers configured to selectively delay a display of thetwo-dimensional OCT view to match the respective latency of thestereoscopic visualization camera; and the controller includes a secondset of image buffers configured to selectively delay the display of thelive two-dimensional stereoscopic view to match the respective latencyof the OCT module.
 12. The system of claim 1, wherein the controller isconfigured to: add at least one annotation over the shared compositeview on a display, the at least one annotation indicating a landmark ofthe eye; and to maintain a relative position of the at least oneannotation in the live two-dimensional stereoscopic view.
 13. The systemof claim 1, wherein registering the first set of volumetric data fromthe OCT module with the second set of volumetric data from thestereoscopic visualization camera includes: aligning the first andsecond views of the stereoscopic visualization camera respectively inrotation, translation and scale to the volumetric render module; andmatching respective perspectives of the first and second views of thestereoscopic visualization camera to the volumetric render module. 14.The system of claim 1, wherein registering the first set of volumetricdata with the second set of volumetric data includes: finding arespective location and respective orientation of a center of projectionof a first two-dimensional visualization module and a secondtwo-dimensional visualization module of the stereoscopic visualizationcamera relative to the respective location and the respectiveorientation of a respective data space of the OCT module.
 15. The systemof claim 1, wherein registering the first set of volumetric data withthe second set of volumetric data includes: aligning a local area ofinterest in the first set of volumetric data in position, orientationand size with the second set of volumetric data; and wherein the localarea of interest includes at least one of a corneal limbus and a scleralvasculature.
 16. The system of claim 1, wherein prior to registering thefirst set of volumetric data with the second set of volumetric data, thecontroller is configured to: calibrate the OCT module and calibrate thestereoscopic visualization camera, including placing a calibrationdevice at the target site and fitting respective lines to respectivesurfaces of the calibration device in each of three orthogonal views.17. The system of claim 1, further comprising: an oblique visualizationmodule selectively executable by the controller to produce an off-axisview of the shared composite view such that a respective representationof the first set of volumetric data and the second set of volumetricdata is viewed at a predefined oblique angle; and wherein execution ofthe oblique visualization module causes the controller to: form avirtual image based in part on the first set of volumetric data; renderthe virtual image as a first pair of stereo images; render the secondset of volumetric data as a second pair of stereo images; and fuse thefirst pair of stereo images and the second pair of stereo images to formthe off-axis view.
 18. The system of claim 17, wherein: the predefinedoblique angle is selectable by a user via a user interface.
 19. A systemfor guiding an ophthalmic procedure, the system comprising: a housingassembly having a head unit configured to be at least partially directedtowards a target site in an eye; an optical coherence tomography (OCT)module at least partially located in the head unit and configured toobtain a first set of volumetric data of the target site; avisualization camera at least partially located in the head unit andconfigured to obtain a second set of two-dimensional image data of thetarget site, including first and second views of the target site; acontroller in communication with the visualization camera and the OCTmodule; a volumetric render module selectively executable by thecontroller; wherein the controller has a processor and tangible,non-transitory memory on which instructions are recorded, execution ofthe instructions causing the controller to: register the first set ofvolumetric data from the OCT module with the second set oftwo-dimensional image data from the visualization camera to create athird set of registered volumetric data; render the third set ofregistered volumetric data to a first region to obtain amulti-dimensional OCT view, via the volumetric render module; render thesecond set of two-dimensional image data from the visualization camerato a second region to obtain a live multi-dimensional view, via thevolumetric render module; and overlay the first region and the secondregion to obtain a shared composite view of the target site.